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This report sunsnarizes the first year evaluation of the 
Cooperative Demonstration Program (High Technology). The Program is 
authorized under Title IV, Part B, Subpart 1 of the Carl Perkins Act of 
1984 and is administered by the Office of Vocational and Adult 
Education. The purpose of the evaluation is to determine the extent to 
which: 1) the grant applications present a clear and coherent design 
for a project, 2) the project designs have been successfully 
implemented; and 3) project costs are reasonable in relation to 
projected or actual outcomes. 

There are three sets of activities that will be conducted during 
the evaluation: a review of grantee applications for 53 projects 
funded in FY1988 or FY1989» mail or telephone surveys of 39 of those 
funded projects, and site visits to 27 of the funded projects. The 
data collected will be presented in a final report addressing the above 
evaluation issues. This evaluation is being conducted during the 
period March 13, 1990 through December 31, 1991. 

Tl.is document is submitted as Deliverable No. 27 to the U.S. 
Department of Education, Office of Planning, Budget and Evaluation 
(QPBE), under Contract No. ':900O60Ol to COSMOS Corporation. Dr. 
Sandra Furey, OPBE, serves as the COTR for the contract. The study 
team would like to gratefully acknowledge the guidance and support 
provided by Dr. Furey during the study's first year. 

The study team consists of staff members from COSMOS Corporation 
and Westat Corporation. Participating in the evaluation in the first 
year from COSMOS were: Peter Bateraan (project director), Ellen 
Schiller, June Sivilli, Chris Owner, and Judith Alamprese (corporate 
reviewer). Participating in the evaluation from Westat (and Decision 
Resources Corporation) were: Lana Muraskin (deputy project director), 
Ted Murphy, Allison Henderson, Janie Funkhouser, and Justin Boesel. 
Three outside experts in the field, Bill Morrill, Gene Bottoms, and 
Roger Vaughan, are helping to guide the evaluation. The study team 
would like to especially thank Roger Vaughan and Lynne Adduci for their 
assistance in editing this report. The report was typsd and assembled 
by Tina Jackson 
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EXECUTIVE SUMMARY 

This report summarizes the activities of and the findings from an 
evaluation of 23 projects funded by the Cooperative Demonstration 
Program {High Technology) in FY1988. The evaluation answers three 
questions about the FY1988 projects. First, did the grant applications 
submitted for funding present a clear and coherent design for the 
project? Second, were the grantees able to ia^lewent their project 
designs as proposed, and, if not, what problems prevented that 
iiBpiefnentation? Third, were the costs of the project reasonable 
relative to projected or actual outcon^s from the project? Each of 
these questions is addressed in a separate section of this report. 

The Cooperative Demonstration Prograw (PL 98-524, Title IV, Part 
B, Subpart 1) is the largest deiwjnstration effort currently supported 
under the Carl Perkins Act of 1984. The Program provides the U.S. 
Department of Education (ED) an opportunity to try new approaches to 
vocational education and to learn about the effectiveness of those 
approaches. The funded projects are to reflect the major priorities of 
the Act: 1) Increased access to high quality programs for special 
populations (disadvantaged and handicapped); 2) improvement of the 
transition from school to wrk for all students; and 3) possibility of 
replication (Federal Register, 1988). The projects also are to 
deJBonstrate successful cooperation between the private sector and 
public agencies to impart advanced vccational education skills through 
a variety of models, e.g., work experience and apprenticeship, worksite 
training, placement, and public works. 

Of the 181 applications submitted In the first year of the Program 
(FY1988), 36 applications were approved and were given grants ranging 
from $50,000 to $550,000. The winning projects were notified in 
October 1988 and began operating as early as December 1988. Although 
operating in calendar year 1989 and 1990, the first cohort of projects 
are considered FY1988 projects because the funding was appropriated In 
the FY1988 budget. The second cohort of projects (a total of 30) was 
funded from FY1989 funds and operated in 1990 and 1991. 
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Not all of the projects funded FY1988 responded to the Secretary's 
priority of addressing high technology Issues. Of the 36 grants 
awarded, the Office of Vocational and Adult Education (OVAE), U.S. 
Department of Education, has Identified 23 as be^ng "high technology," 
based on either the tvoe of lob for which traininc; is offered (or 
curriculum developed) or the nature of the training being given the 
student. These 23 projects are the focus of the first year evaluations 
in this study. 



L An Assessment of Proi ect Logic and Des ign 

Ideally, the Program should fund only those applications that 
offer a definable treatment and the possibility of success. The 
identification of the discrete services and the determination of the 
likelihood for ;uccess of the proposed projects can be subjected to an 
evaluability as;essment (EA). EAs traditionally determine how best to 
design a program evaluation, but the logic and plausibility components 
of the methodology can also be used to determine the clarity and 
coherence of individual projects undertaken as part of that overall 
program. The partial EA of the 23 FY1988 high technology projects 
determined the "clarity" and "coherence* of the designs of the funded 
projects. 

Project logic and plausibility are both important predictors of 
project success and of the ability to measure that success. Eight 
projects were ranked high, ten projects were ranked medium, and five 
projects were ranked low in logic. Four projects ranked high, 12 
projects ranked medium, and seven projects ranf^eu low in plausibility. 
Most project:, received a different ranking on logic than they did on 
plausibil ity. 

To anticipate success, a project must be ranked highly on both 
logic and plausibility. This combined rating suggests that the project 
will be likely to accomplish the outcomes and activities stated in the 
application. That is, the project will have a greater likelihood for 
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demonstrating that it can achieve Its stated goals. In contrast, 
projects rated low In either plausibility or logic are predicted to be 
less successful. Projects receiving raid-range ratings In either logic, 
plausibility, or both, are predicted to be only somewhat successful. 
When taking Into account both factors, three projects were predicted 
successful; ten were predicted somewhat successful; and ten projects 
were predicted less likely to be successful. 

The predicted success of the project based on the application's 
clarity and coherence can be compared with the actual success of the 
project based on the numbers of students trained. Among the three 
projects most likely to succeed, two projects (Valencia [mfg.] and 
Richland) fell short of their targeted number of students to be 
trained. Among the two projects (Skyline and Valencia [film]) somewhat 
likely to succeed, both projects met their training target, although 
Skyline had a much lower participation rate that initially expected. 
Among the three projects less likely to succeed, two projects 
(University of Wisconsin-Stout and Northampton) fell short of their 
targeted number of students to be trained. It appears that the clarity 
and coherence of the application is not a good predictor of the likely 
success of the training component of the project. Other aspects of the 
project, e.g., long term impacts on the labor market or numbers of 
students getting jobs because of the training could not be evaluated 
within the time-frame of the current study. Therefore, the clarity and 
coherence of the application as predictors of long terra success is 
still unknown. 
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Successful Implementation meant that projects both carried out the 
activities proposed in the original gr^nt application and accoaplished 
the objectives of the project. However, the Department of Education 
can gain important lessons in the practicality of the project design 
and the operation of the demonstration program as a whole from projects 
that failed in one or both of these considerations. 

L Project Administration 

Timing of Award : All but one of the projects visited began 
spending funds in January 1989, but most Indicated that the timing of 
award notification was d'*'*"icult for theiB. The main problem was that 
most of the grantees wer». dcademic institutions operating on a 9-month 
academic year beginning in September. 8y the time the grants started, 
almost half their potential "planning" year was over. Grantees that 
were already involved in activities for which Federal support was 
received were better able to accommodate the mid-year start date. 

Hiring Staff : Most projects did not plan to hire new staff in key 
positions, but of those that planned to do so, few did. The timing of 
award and the IS-month time frame were the main problems, i.e., 
recruiting and hiring took sever<il months and could not always be 
completed. These projects had difficulty either because no qualified 
candidate would take the position for only 18 months (or less by the 
time the recruitment procedures were fulfilled) or because the grantee 
did not want the responsibility for employing the staff after the grant 
ended. In several projects, responsibilities among staff members were 
shifted during the project and no permanent staffing "groove" was 
created bevore the grant was ended. 

Purchasing Equipment : Most grantees perceived that grant funds 
could not be used to purchase equipment, although there was no outright 
restriction on equipment purchases in the FY1988 grant awards. The 
perception of the restriction probably led to fewer equipment 
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purchases. However, several projects expended considerable staff 
effort to find other sources of funding for equipment. 

Establishing Private Sec tor Partnerships: The term "partner" had 
different meanings among projects. In most cases, the "partner* was a 
private employer enlisted by a public grantee. Most partners played 
limited and conventional roles: 1) offering one-time or periodic 
advice on curriculum design; 2) serving on projects* advisory 
committees; or 3) identifying marketable skills. Several projects also 
obtained donated equipment or other materials from local businesses, 
sometimes in return for training their employees. 

Three of the eight projects visited were essentially customized 
training programs for employees of "partner" companies, and one planned 
to offer no-cost employer- specific training but ran out of time. Two 
projects provided broader training programs— one for people seeking 
employment in Toyota auto dealerships and one for potential employees 
in Universal Studios, but neither offered guaranteed employment. Two 
projects offered one-year introductory programs for high school 
students in particular high-tech fields but targeted no specific 
employers. Two projects had more extensive involvement of the private 
iector— in fact, were initiated by private employers. Some projects 
also established, or enhanced, "partnerships" between secondary school 
districts and postsecondary institutions. 

Other Start-UD Activities : At the outset, the assessment 
Identified several activities likely to occur during the project start- 
up period, including recruiting students, adapting curricula, 
establishing management information systems, obtaining necessary 
approval and accredition for course offerings, identifying job 
opportunities, and conducting public relations activities. Some 
projects engaged in elaborate student recruitment efforts. Depending 
on the participants the projects planned to attract students already 
enrolled in secondary or postsecondary institutions, cr persons who 
were not in school. Projects recruiting new, less-advantaged students 
invested heavily in recruitment. Those training current employees of a 
single "partner" employer or students already enrolled in the grantee 
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Institution recruited less. One project encountered start-up problems 
when a private sector partner failed to provide promised training 
placements for graduates. Another project found planned access to 
private employers through an economic development agency no longer 
viable and was forced to forge new links. 

2. A Summary of Project Content and Intermediate Qutcoines 

The projects studied exhibited four different emphases, each with 
different likely outcomes: 1) provision of technical training with the 
aim of job placement or upgraded jobs, 2) Identification of the skills 
needed in particular occupations and development of appropriate 
curricula, 3) testing of existing programs to serve a wider set of 
Industry needs, and 4) inservice training of faculty aimed at 
developing new courses. Some projects emphasized only one area while 
others aimed at two or more emphases. 

Direct Training Services to Students : With one exception, all the 
visited projects emphasized the vocational or technical training of 
students, although the amount and duration of training varied. Some 
provided extensive training. For example. Skyline Community College 
provided each student 16 weeks of full-time automotive mechanics 
training per year for two years and planned to offer a third year for 
students to complete the program. Francis Tuttle provided more than 
2,000 hours of training over 24 months. On the other hand, 
Northampton's training modules in surface mount technology offered only 
a few days training. In some extensive training projects, i.e., 
Francis Tuttle and Moorpark, only a part of the training was supported 
by the Federal grant, with grantee institutions or the partner 
organization providing the remaining funds. The role of training also 
varied. In two cases, training was provided in order to test new 
curricula: the main goal at Richland was to disseminate a new 
curriculum and train 250 students in materials technology and at 
Northampton training was used to test and refine curricula. These 
differences in goals and intentions make cross-site comparisons of the 
level and duration of training difficult. 
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None of the projects offered graduates a guarantee of a new job or 
a promotion; however, several said that these outcomes were likely. 
Several projects provided customized training to persons who were 
alteady eroployees (or trainees) of private companies. 

Most visited projects provided short-term training separate from 
regular offerings, although a few tried to add new components to 
ongoing programs. Most projects were "add-ons" to regular work of the 
Institutions— short-term training sessions at community colleges rdtner 
than new courses. The emphasis on customized training indicates the 
short-term focus~a product of the short-term Federal funding . 

Curriculum Oevelopment! A secondary goal of some projects was the 
development of new curricula. All the projects except Indiana 
University engaged in a formal process to develop new curriculum and at 
four projects curriculum development was a major focus of the grant. 
Despite emphasis on curriculum development, few assessed the curricula 
they developed. Only a few of the projects systematically sought 
teacher feedback and adjusted programs accordingly. Only Richland 
reviewed curriculum systematically, prior to introducing the curriculum 
and none established means to determine the effectiveness of the 
curriculum in the classroom. One project simply packaged and 
distributed the "modules" developed by teachers who participated in an 
inservice training workshop. 

Skills Identif ic-* t;on: A number of projects proposed to 
demonstrate how to identify vocational and/or academic skills needed 
for employment in particular occupations, usually for those facing a 
shortage of qualified workers. But visited projects had developed 
systematic ways to identify needed job skills. Local business 
representatives on project advisory councils were often consulted about 
what they thought was needed. Several projects contacted local 
employers, not necessarily project partners, to find out what skills 
they wanted for employees or potential employees. Of the nine 
projects, one project surveyed local businesses and industries, and 
anoth' r conducted a literature review to identify skills. 
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Staff Developisent : A few o* the projects emphasized staff 
development— probably because of the demonstration nature of the grants 
(staff development tends to be an activity with an inherently local 
focus). Two projects, Richland and University of Wlsconsln-St^ut, 
heavily emphasized staff development. They each brought together 
secondary and postsecondary Instructors and helped thera learn how to 
construct training modules in new fields or adapt develop new materials 
and experiments for use in the classroom. These projects effectively 
"tested" a model in which teachers develop curricula from information 
provided in workshops. 

Nonetheless, most projects did train staff, although the level 
varied. Three projects introducing new curricula intensivel.v trained 
teachers. One project trained staff to assess students* basic skills 
and tauo^t them reading and math skills in a learning lab. Two 
projects already have staff development programs prior to receiving the 
Federal grant. Others (Valencia [film], Valencia [mfg.], c.id Indiana 
University) conducted no formal staff development, but allowed informal 
teacher training or encouraged instructors to spend their own time Irr 
curriculum development. 

All projects planed to disseminate their findings, but only one 
made dissemination a major emphasis. Project directors at other sites 
made presentations at regional (or national) vocational education 
conferences and sprf?ad Information to community colleges via 
association meetings and conferences. 

3. Innovation Among Projects 

The purpose of a demonstration can be to either 1) pi-ove that an 
idea of concept works, much like an experiment, or 2) show how 
something works or p rforms, much like a presentation. Whereas the 
first purpose implies the development and testing of new ideas, the 
latter purpose stresses replication of existing ideas in new contexts. 
Program regulations for the Cooperative Demonstration Program do not 
specify which of the two (or both) purposes is to be addressed by the 
grantees and applicants are free to select which emphasis they wish to 
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pursue. The study team looked at two aspects of projects vhich may 
reflect the testing of new ideas; the high techiiology aspect of each 
project and the Innovation, If any. In vocational sd'icat^on services. 

The "high technology" characteristics of the nine Cooperative 
Demonstration Grant projects varied. They included: 

■ T-^aining to prepare studsnis for jobs <n 
fields thU manufacture high technology 
products; 

■ Training to enable studencs to use high tech 
equipment, in fields not usually considered 
to be high tech; 

■ Training in the use of computers, regardless 
of the field for which training is received; 
or 

■ Training in basic skills to prepare graduates 
for specific occupational training in high 
tech fields. 

Although the FY1988 program regulation* and application materials 
do not require projects to be innovative (nor is the term "innovative" 
even used), the study *s Advisory Panel and staff felt that it was 
important for the demonstrations to advance the knowledge of the 
vor=»tional education field by testing new ideas. At each project 
visited, the study team sought to understand what, if anything, the 
project saw as innovative or new in its offering. Much of what was 
observed during the site visit was hardly "cutting edge" with respect 
to instruction, partnership, supplementary or coordinated services, or 
serving special populations {as defined by the Perkins Act or 
otherwise) . 

The absence of innovation within and across projects might have 
been due to the absence of a priority for it in the regulations or 
points awarded for it in the scoring of applications. Lack of 
innovation might also be due to the short time frame of the grant— it 
1s hard to start and complete an innovative project in 18 months. 
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Still, a few projects did manage to experiment with new designs, 
services, or partnerships. 

Most projects met the needs of those they served— opening job 
opportunities in growing fields, upgrading workers* skills, exposing 
small firms to the opportunities offered by high technology equipment, 
and increasing the capacities of training institutions and staffs to 
train people in high technology fields. When asked who benefitted from 
the projects, project staff cited three beneficiaries: trainees, 
employers, and the institutions providing the training. 

Finally, partner roles in project activities were limited and 
several projects failed to develop promised partnerships or failed to 
live up to expectations. In at least three projects (Skyline, Indiana 
University, and Moorpark) the private sector partners appear to have 
made inflated promises about what they could deliver. When forced to 
deliver job placements, employers seeking assistance, or support 
services, were unable to do so. Failure to deliver put these projects 
at risk. 
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C. Project r.Qsts and Benefits 

The third question Is whether project costs are "reasonable" In 
relation to tha projected or actual outcomes of the project. Project 
costs are defined as the Cooperative Demonstration Program grant plus 
the non-Federal cash and/or in-kind matching coritributions provided by 
the grantees. The outcomes of the project are defined as the numbers 
of students trained, the number of staff trained (if the project also 
focused on staff development), and the number of course hours developed 
(if the project also focused on curriculum development). 
"Reasonableness" compares costs and benefits among projects rather than 
comparing them to some absolute standard. No attempt was made to 
assign a monetary value to the benefits resulting from the project 
outcomes (e.g., the dollar value to the student for having learned the 
new skill), but compares project treatment costs and outcomes. Average 
project costs and outcomes were aggregated to allow an overall cost 
estimate and benefit estimate. Cooperative Demonstration Program 
grants, however, require that the grantee contribute at least 25 per 
cent of the total project costs. Thus, project costs consist of both 
the Cooperative Demonstration Program grant amount and the grantee 
match. Project costs, therefore, are the sum of all resources- 
financial and non-financial— of accomplishing proposed and/or actual 
activities. 

Total project costs ranged from $308,335 for Northampton Community 
College to $759,842 for Richland Public Schools. Only one project, 
Skyline Community College, received other grants that contributed 
directly to the Cooperat ve Demonstration Program project but which 
were not reported as part of the local match. The percent of total 
project costs contributed by the grantee ranged from 26.1 percent by 
Northampton Conwunity College to 57.6 by Richland Public Schools. The 
district with the lowest total project costs also contributed the 
lowest while the district with the highest total project costs 
contributed the highest local match. 
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Project benefits are the outcomes that improve the quality of the 
vocational -education process. These can be quantifiable (e.g., the 
number of students successfully completing training) or non- 
quantifiable (e.g., a new technique for skills identification). Non- 
quantifiable outcomes are difficult to measure and to express in terms 
that would allow comparison among projects. Consequently, the cost 
benefit analyses focus only on quantifiable outcomes— those activltier 
for which cost data were available (either through the grantees 
accounting system) or through the final grant budget. Three activities 
were measured through costs data or budget allocations: planning and 
administration; student training; and curriculum development. Planning 
and administration costs include: the wages of the project director 
and clerical staff, associated fringe benefits, other direct costs 
associated with administration, and Indirect costs. Student training 
costs include: the wages of instructors and other specialists, 
associated fringe benefits, other direct costs ^e.g., textbooks, 
supplies, travel, and stipends), and indirect costs. Curriculum 
development costs include: the wages of instructors and curriculum 
development specialists, associated fringe benefits, other direct costs 
(e.g., training workshops, travel, and printing), and indirect costs. 

The separate cost and benefit factors were used to address four 
issues. The proportion of total project resources devoted to services 
instead of administration ranged from a low of 64.8 percent in Moorpark 
to a high of 97.6 percent In Richland. The costs for Richland appear 
artificially low because it does not include the cost of administration 
in partne»* schools. Second, the amount of training per student 
completer ranged from 2.3 hours at the University of Wisconsin-Stout to 
1,280 hours at Skyline Community College. Staff training ranged from 
30 hours per teacher to 160 hours per teacher. Third, the average unit 
cost for student training ranged from a low of $0 at the University of 
Wisconsin-Stout (all training costs were born by the participating 
schools as part of their regular teaching responsibilities) to a high 
of $25.53 at Indiana University, 
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It should be noted that comparing the average cost per hour of 
training across projiKts may create an unfair comparison because of the 
variations in the intensity of the training and the number of students 
being trained. The average cost for an hour of training at Indiana 
University was the highest of the projects, but that project also 
trained the second highest number of students. To provide a more 
accurate comparison, the analysis should compute the average cost per 
hour of training per student trained . The per hour per student costs 
range from a low of $0.00 for the University of Wisconsin to a high of 
$.89 at Sl^yline Cofffliunity College. The costs for the remainder of the 
projects tend to concentrate between $.01 and $.06 per hour. Skyline's 
costs are out of proportion with the other projects because so few 
students were hired into the program by the Toyota dealers and the 
program operated at a level far lower than originally planned. 

Fourth, the total treatment cost was calculated as the sum of all 
costs associated with providing the program. Service costs ranged from 
a low of $0 at University of Wisconsin to $19,266 at Skyline Community 
College. (Skyline's costs are out of proportion with the other 
projects because the project trained fewer students than originally 
pi anned.) 

The answer to the question "are project costs reasonable in 
relation to project outcomes?" appears to be yes for all project except 
Skyline. The per unit and per outcome costs for other projects were 
similar even though total costs and project intensity varied 
substantially. Although the reasons for Skyline's costs were 
understandably high, their results suggest that the project was 
unsuccessful as a model for other vocational education institutions. 
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INTRODUCTION 



This report sunmarlzes the activities and the findings of an 
evaluation of 23 projects funded by the Cooperative Demonstration 
Program (High Technology) In FY1988. The evaluation answers three 
important questions about the projects. First, did the grant 
applications submitted for funding present a clear and coherent design 
for a project? Second, were the grantees able tc implenjent their 
project designs as p.'oposed, and, if not, what probletns prevented that 
impleflientatlon? Third, were project costs reasonable relative to 
projected or actual outcomes? Each question Is addressed in a separate 
section. 

Overview of the Cooperative Demonstration Program 

The Cooperative Demonstration Program (PL 98-524, Title IV. Part 
B, Subpart 1) Is the largest demonstration effort currently supported 
under the Carl D. Perkins Vocational Education Act of 1984. The 
Program provides the U.S. Department of Education (ED) and educational 
Institutions an opportunity to try new approaches to vocational 
education and to learn about the effectiveness of those approaches. 
Funded projects reflect the Act's priorities: Increased access to high 
quality programs for special populations and the overall improvemen* of 
the quality of vocational education. Projects are also to domonstrate 
successful cooperation among private employers and public age.icies that 
results In training In advanced vocational education skills. A variety 
of models are suggested, Including: '♦<ork experience and apprentice- 
ship, worksite training, placement, and public works. Agencies 
eligible to apply include state education agencies (SEAs), local 
education agencies (LEAs), pnstsecondary educational institutions, 
Institutions of higher education, and other public and private 
agencies, organizations, and Institutions, 

Projects may be funded through grants, cooperative agreements, or 
contracts and may be: 



Model projects providing improved access to 
quality vocational education programs for— 

-handicapped Individuals; 

-disadvantaged individuals; 

-adults who are in need of training 
and retraining; 

-Individuals who are single parents 
or howeiuakers ; 

-individuals who participate in programs 
designed to eliminate sex bias and 
stereotyping in vocational education; 

-criminal offenders who are serving 
in a correctional institution; and 

-men and women seeking nontraditional 
occupations. 

Projects that are examples of successful 
cooperation between the private sector 
(including employers, consortia or 
employers, labor organizations, and 
building trade councils) and public 
agencies In vocational education, including 
Statu boards and eligible recipients. The 
projects must be designed to demonstrate 
ways in which vocational education and the 
private sector of the economy can work 
together effectively to assist vocational 
education students to at:ain tne advanced 
levul of skills needed to make the 
transit iv*^ from school to productive 
employment, including 

-work experience and apprenticeship 
projects; transitional worksite job 
training for vocational education students 
which is related to their occupational 
goals and closely linked to classroom and 
laboratory instruction provided by an 
eligible recipient; 

-placement services in occupations which 
the students are preparing to enter; and 



-where practical, projects that will 
benefit the public, such as the rehabili- 
tation of public schools or housing in 
inner cities or economically depressed 
rural areas. 

The projects may include Institutional and 
on-the-job training, support services 
authorized by Ihe Act, and such other 
necessary assistance as the Secretary 
determines to be necessary for the 
successful completion of the project. 

3. Projects to overcome national skill 
shortages, as designated by the Secretary 
in cooperation with the Secretary of Labor, 
Secretary of Defense, and Secretary of 
Conwierce. 

4. Such other activities which the Secretary 
may designate which are related to the 
purposes of the Act. [Federal Register, 
1985 pp. 33260-33261.] 



All projects, however, must directly serve people enrolled in 
vocational programs and be widely replicable by service providers. 
Furthermore, grant recipients must provide, through cash or Inkind 
contributions, a minimuai of 25 percent of the total cost of the 
demonstration project. The contributions can include the fair market 
value of facilities, overhead, personnel, and equipment. 

In addition to the priorities contained in the Act, the Secretary 
of Education has the prerogative each year for establishing additional 
priorities for the Program. In the first year of the Program (Fyi988), 
an invitational priority was issued for projects addressing high 
technology (in FY1989 this became an absolute priority). The term 
"high technology" was defined to mean: 



state-of-the-art computer, microelectronic, 
hydraulic, pneumatic, laser, nuclear, chemical, 
telecommunication, and other technologies being 
used to enhance productivity in manufacturing, 
coranunlcatlon, transportation, agriculture, mining, 
energy, conanercial, and similar economic activity, 



and to improve the provision of health care [34 CFR 
400.4(b)]. 

The ApDlication Process 

The projects funded by the Program were proposed and implemented 
by educational institutions, private agencies, and other organizations. 
Each year since the Program began in FY1988, the Office of Vocational 
anu Adult Education (OVAE), U.S. Department of Education, has solicited 
grant applications from agencies or organizations Interested in 
conducting demonstrations. Program guidelines, along with instructions 
for submitting applications, were published In the Federal Register and 
mailed directly to perspective applicants. 

In general, the application process was as follows. Applicants 
prepared and submitted project applications according to the published 
guidelines. Applications were reviewed by OVAE staff and internal 
reviewers. Applicants were asked to clarify any unclear aspects of 
their applications. The applications were judged according to the 
following selection criteria and point allocations: 

■ Statement of need (15 points); 
B Plan of operation (30 points); 

■ Quality of key personnel (10 points); 

■ Budget and cost effe'-tlveness (10 points); 

■ Evaluation plan (5 points); 

■ Adequacy of resources (5 points); 

■ Private sector involvement (10 points); 

■ Employment opportunities (5 points); and 
• Dissemination (10 points). 

The projects receiving the top scores were awarded grants. As many 
projects were funded as the program budget would allow. In FY1988, a 
total of $9.5 million was awarded. 



Of 181 applications submitted In the first year of the Program 
(FY1988), 36 wre approved and giver, grants ranging in size from 
$50,000 to $550,000. The winning projects were notified in October 
1988 and began operation as early as Oecenber 1988. Although operating 
in calendar year 1989 and 1990, the first cohort of projects are 
considered FY1988 projects because the funding was appropriated frcm 
Fyi988 Perkins Act funds. (The second cohort of projects— a total of 
30— was funded from FY 1989 funds and began operation around January 
1990.) 

Not all projects funded in FY1988 had responded to the Secretary's 
invitational priority of addressing high-technology issues. Of the 36 
grants awarded, OVAE identified 23 as being "high technology/ based on 
either the type of job for which training was conducted (or curriculum 
developed) or the nature of the training given students. These 23 
projects were the focus of the first year evaluation effort. The list 
of the Fyi988 high technology projects is presented In Exhibit 1. 

Defining Project Boundaries 

For purposes of this evaluation, the "project" is defined as those 
activities funded by the Cooperative Demonstration Program grant and 
matching funds and occuring within the 18-month grant period. Services 
provided before or after the grant period and benefits accruing after 
the end of the grant are not included In the present scope of inquiry. 
Also not included is the underlying training and support services 
offered to project enrol lees. For example, a project participant might 
enroll in training developed with project funds and provided by a 
project-supported instructor, using equipment donated by a local 
business as an official project "match." Yet to complete the degree or 
certificate for which the training was developed, the student may be 
expected (by the project design) to enroll in additional Instruction in 
the same institution in courses not supported by the project. Students 
might also use financial aid or support services, provided by or 
through the Institution, that were n. t part of the project. All 



Exhibit 1 

GRANTEES WITH HIGH TECHNOLOGY PROJECTS 



Grantee 



Division of Vocational/Edu- 
cation Services 
State Department of Education 
Montgomery, Alabama 

Skyline Conege 

San Bruno, California 



Ventura Community College District 
Moorpark College 
Moorpark, California 

Valencia Community College 
Orlando, Florida 



Valencia Coirenunity College 
Orlando, Florida 

Parkland College 
Champaign, Illinois 

Waubonsee Conanunity College 
Sugar Grove, Illinois 

Indian Hills Coisnunity College 
Ottumwa, Iowa 

Hampden County Employment 
Training Consortium 
Springfield, Massachusetts 

Central Community College- 
Platte Campus 



Postsecondary Vocational - 

Technical Education 
Concord, New Hampshire 



Project 



Student Apprenticeship link- 
age in Vocational Education 



Toyota/Skyline Partnership 
for Automotive Technician 
Training 

Non-college Bound Student 
Demonstration Project - Elec- 
tronics/Laser/Electro-optics 

A Model, Replicable Advanced 
Manufacturing Demonstration 
Project 

Film Production Technology 
Training Program 

Advanced Certification Program 
for Computer Graphic Specialists 

A Comprehensive Development Plan 
in Office Skills 

Indian Hills Cooperative Demon- 
stration Program 

Project CREATE: Cooperative Re-and 
sources to Enhance Access to 
Jobs Through Technical Education 

Competency-Based Modular Assess- 
ment and Training for Maintenance 
Technicians in Manufacturing 

New Hampshire Automotive 
Education Collaborative 



Exhibit 1, (continued) 
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Grantee 



Southern Growth Policies 
Board 

Research Triangle Park» 
North Carolina 

University of North Dakota- 
lake Region 

Devils Lake, North Dakota 

Toledo Public Schools 
Toledo, Ohio 

Francis Tuttle Vocational 
Technical Center 
Oklahoma City, Oklahoma 

Portland Community College 
Portland, Oregon 



Northampton Community College 
Bethlehem, Pennsylvania 

Indiana University of 

Pennsylvania 

Reschini House Indiana, 

Pennsylvania 

Greenville Technical College 
Greenville, South Carolina 

El Paso Conwnunity College 
El Paso, Texas 



Richland School District 
Kennewick, Washington 



Project 



Consortium for Manufacturing 
Competitiveness 



Flight Simulator Maintenance 
Technician 



Industrial Automation Mechanic 
Model Curriculum 

High Technology Partnership 
Project 



Women in Education for Appren- 
ticeship and Non-Traditional 
Employment 

Turn-key Surface Mount 
Training Progr<im 

Northwestern Pennsylvania 
Cooperative Demonstration for 
Technical Updating 



Project TEAM: Technical 
Education Advancement Modules 

CAREER Program: Career Assessment, 
Remediation, Education, Employment, 
and Re-entry 

Materials Technology: The Common 
Core Skills That Are Shaping the 
Future 
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Grantee Project 



Yakima Valley Community Extending Health Training and 

Yakima, Washington Services to Rurally Isolated 

Populations in a Depressed Area 

University of Wisconsin- Imp] eroen ting a High-Tech Train- 

Stout ing Model for Rural Based Busi- 

Menomonie, Wisconsin ness and Industry, Technical 

Colleges, and local and State 
Education Agencies 
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components might be necessary to complete the degree or certificate, 
but only those directly funded by the grant are included In the present 
evaluation. 

The boundaries of the project do include the interorganizational 
networks that developed in the course of carrying out a project* s 
mission. Grantee institutions entered into a variety of formal and 
informal relationships with other organizations to offer students 
support services, provide them with jobs, or provide the project with 
additional financial or other assistance. 

In its June 1990 meeting, the evaluation's Advisory Panel 
expressed three concerns about the definition of a project. First, 
they felt that 18 months was too short for a truly innovative project- 
or perhaps any project— to show results. The limited time frame may 
have made the Program less attractive to secondary (but not post- 
secondary) institutions, resulting in fewer applicants. Second, the 
Panel felt that limiting the evaluation to the grant period did not 
allow the history of the projec. or institution to be taken into 
account. Projects may be a part of a larger on-going efforts by 
grantees, and the important effects will be from the larger effort 
rather than the one part. Third, the Panel did not want to limit the 
evaluation to what was accomplished with Federal funds. They felt the 
study asked only whether the Federal money made a difference rather 
than whether the overall project (of which the Federal funds supported 
a part) was able do new things and help students. Furthermore, the 
study should determine ^f the project would have started without 
Federal funding. Limiting the scope to the Federal funding and direct 
local match made it difficult to gauge the importance of the grant 
within a larger institutional effort. 

The Panel recomnended expanding the evaluation period beyond 18 
months to Include both prior grantee activities and effects after the 
grant period. The study team should select the most Interesting 
projects from the first round of grants (e.g., University of 
Wisconsin - Stout, Richland School District, and Moorpark College) and 
track them after the Federal grants end. This would not necessarily 
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mean a secord site visit; additional participant and outcome data could 
be collected through mail and telephone survey. These recoimnendations 
have been incorporated in evaluation design and definition of project 
boundaries for Year 2. 

Overview of this Report 

The first year report Is presented in five sections. Section I 
presents an introduction to the Cooperative Demonstration Program and 
the three major questions to be addressed in the evaluation. Section 
II describes the modified, evaluability assessment done of the 23 
FY1988 projects and answers the first study question regarding the 
cle.r and coherent design of the projects. Section III describes the 
implementation experience of a subset of FY1988 projects as determined 
through a telephone survey of nine projects and site visits to eight of 
those nine projects. Section IV analyzes project costs relative to 
project accomplishments and answers the third study question regarding 
the "reaso,-»ableness" of project costs. Section V summarizes the 
findings of the first four sections and recom:.iends potential 
improvements in the program. 
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11. AN ASSESSMENT OF PROJECT LOGIC AND DESIGN 



Aplications for Cooperative Oeinonstration Program grants are 
submitted to the U.S. Department of Education where the applications 
are reviewed and ranked by a panel of experts. The review and award 
process Is Intended to accomplish two objectives. First, It identifies 
applications capable of achieving the purpose of the demonstration 
program— the design of new program or the replication of a validated, 
existing program. Second, the process Identifies applications 
describing projects likely to succeed— to accomplish project objectives 
In a timely and measurable manner. 

Projects wh1c^ are clear and coherent in their design are more 
likely to succeed. Although applications are not awarded specific 
points for clarity and coherence, these characteristics are important 
to the success of the project and its ability to yield measurable 
outcomes. Applicants are requested to submit a "concise and clearly 
written Program Narrative" that includes: 

■ A clear description of the need for the 
proposed project; 

■ A clear statement of what thp project seeks 
to demonstrate; 

■ A clear description of how the objectives of 
the project relate to the purposes of the 
program; and 

■ A clear description of how the applicant will 
provide equal access and treatment for 
eligible project participants. [Federal 
Register, 1985.] 



This section analyzes the extent to which the funded applications 
present a clear and coherent design. 

Because applicants are permitted to be creative, and because there 
are few requirements restricting what applicants may propose to do, the 
re- i ewers face a twofold challenge. First, they must identify what the 
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applicants are proposing to do. The applications may be unclear about 
objectives and intended outcomes. Second, reviewers must determine 
whether the stated oucomes and methods are plausible. Activities and 
outcomes may be clearly stated, but how the former will achieve the 
latter may not be clear— the work proposed may not be reasonable given 
time and resource constraints. 

Ideally, only applications offering a discrete set of services and 
the possibility of success should be funded- Identifying discrete 
services and the likelihood of success can be determined using a 
methodology known as "evaluability assessment" (EA). EAs are 
traditionally used to determine the appropriate design for a program 
evaluation, but the logic and plausibility components of the metho- 
dology can also be useful to determine the clarity and coherence of 
individual projects within a broader program. This section assesses 
the logic and plausibility of the 23 high technology projects funded 
under the Cooperative Demonstration Program in FY1988. This partial EA 
determines the extent to which the designs of the funded projects are 
"clear" and "coherent." 

This section is organized into three parts. Part A describes the 
procedures used to develop a logic model for each project and to score 
the level of project logic. Part B describes the procedures used to 
score the plausibility of the project design. Part C summarizes the 
results of the scoring of project logic and plausibility. 

A. Assessing the Logic of the Pro.iect Design 

Projects are likely to be more successful if applications clearly 
and logically describe the activities to be accomplished and the 
objectives to be achieved. A design may be clear but not logical; 
also, some components may be clear and logical and others may not be. 
Logic requires well-defined objectives (defined in terms of specific 
inputs, activities, and outcomes) and activities linked to those 
outcomes. 
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To determine logic, the study team reviewed each application to 
identify: 

■ A statement of inputs or resources; 
• A statement of activities or events; 

■ A statement of outcomes (both short- and 
long-term); and 

■ A statement of causal links among events and 
outcomes, establishing the expected flow of 
outcomes and results of the project. 

For each of the 23 reviewed applications, the study team recorded 
the components in a logic shell. The shell enabled analysts to display 
in respective columns the links among the components and the existing 
logic of a project. Two contrasting examples of logic models are 
provided. Figure 1 displays a logic model where several activities are 
linked to outcomes, suggesting a clear and logical design. Figure 2 
displays a project with few evident linkages, suggesting a design with 
clearly stated components but with no logical relationships among those 
components. Scoring the logic model determines whether there is a 
logical design, not just an observation of graphically-presented 
project components and linkages. Scoring procedures for the logic 
models are described below. 

Developing the Logic Model 

The study team followed six steps to model the project's logic for 
achieving its proposed outcomes. The model was developed solely with 
information in the grantee's application. First, the team reviewed the 
contents of the project folder and assembled all material pertaining to 
the grant application. Sources were the original application, any 
revisions to the application, correspondence dated orlor to the award 
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Figure 2 

ILLUSTRATIVE LOGIC MODEL WITH FEW LINKAGES 
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date of the grant, grant award documents, and any supplementary 
material submitted by the applicant prior to the date of award. 

Second, the team identified any long-term project outcomes from 
the "Introduction," "Statement of Need," and "Objectives" sections of 
the application. The long-term outcomes were listed on the logic model 
shell. The team recorded the information as stated in the application 
and, if available, specified the target population. Verbs were used to 
begin each of the statements to aid In the visual presentation of the 
logic model . 

Third, the team identified any short-term objectives using Infor- 
mation from the "Plan of Operation* section of the application. These 
were listed on the logic model shell. The team recorded the 
information as stated In the application and specified the target 
population. Again, verbs were used to begin each statement. 

Fourth, the team identified proposed activities from the "Plan of 
Operation" section of the application. The activities were listed on 
the logic model shell. Again, the team recorded the information as 
stated in the application and began the statements with verbs. Fifth, 
the team identified the resources and other Inputs available to the 
project on the first day of the grant award as stated in the "Budget/ 
"Introduction," and elsewhere in the application. The resources were 
listed on the logic model shell. 

Finally, the team prepared a flow model of causal relationships 
among Inputs, activities, short-term goals, and long-term goals. These 
relationships were designated by arrows from one box to another. To 
determine the relationship among Inputs and activities, the team 
reviewed the budget narrative for evidence of how funds, Federal and 
local, were to be spent. A linkage was represented in the flow diagram 
when the narrative corresponded to activities specified on the diagram. 
Other relationships stated in the remainder of the application were 
added in the flow diagram. Since paragraphs represent complete 
thoughts, a linkage was considered explicit only if the two elements 
were expressed within the same complete thought. This Is a restrictive 
requirement that affected final rankings. However, without such a 
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restriction, analysts could not determine whether elements in separate 
paragraphs were intended by the applicant to be connected. Boxes not 
connected by arrows represented gaps in the flow of logic. 

Scoring Prelect Logic 

Using this logic luodel, applications were scored in three steps. 
First, the study team counted the number of inputs, activities, short- 
term outcomes, and long-term outcomes. Second the team determined 
whether there were activities identified for achieving each short-term 
outcome, assigning one point for each short-term outcome supported by 
at least one activity and subtracting one point for each short-term 
outcome not supported by at least one activity. Third, the team 
determined whether there were short-term outcomes identified for 
achieving each long-term outcome. Again, the team assigned one point 
for each long-term outcome supported by at least one short-term outcome 
and subtracted one point for each long-term outcome not supported by at 
least one short-term outcome. The scoring provided the basis for 
analyzing the logic of the project designs. 

The design of the project was considered clear and logical if it 
met five conditions: 



1. It identified one or more long-term 
outcome(s) for the project, e.g., access to 
quality vocational education training or 
making local industry more competitive; 

2. It Identified one or more short-term 
outcomes that would be achieved during the 
grant period and which would help achieve 
the long-term outcome(s); 

3. It specified one or more activities that 
would be implemented during the grant 
period that would help achieve a short-term 
or long-term outcome; 

4. It had more linkages than gaps among 
activities and short-term outcomes—scoring 
positive logic points (a project that has 
four stated linkages and two gaps would 
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have a net score of +2). Otherwise, 
projects would have spent time and money 
just planning; and 

5. It had more linkages than gaps among short- 
term and long-term outcomes. Total points 
must be positive {a project with three 
stated linkages and four gaps would have a 
total score on this criterion of -1). 

RgsuH^ Qf t-hg Uqi<; 

For each grantee, a project model was developed including: 
inputs, activities, short-term outcomes, and long-term outcomes. All 
23 projects addressed these components in their applications. Many 
projects used similar inputs, activities, and outcomes. Table 1 shows 
the common types across the projects. Typical inputs included Federal, 
State, and local dollars, staff already on board (versus hiring new 
staff), and equipment already in operation. Typical long term outcomes 
included "creating a qualified labor pool" and "attracting new business 
to the cotranunity" through a strong economy and labor pool. Projects 
emphasized econoiuic growth, dissemination, and impact of training on 
students and community. 

Description of the Prolect Looic . Using the scoring procedures 
described above, each project was assigned an overall logic score. 
After testing for the presence of all five cr.iditions, it was found 
that: 

■ All 23 projects identified one or more long- 
range outcome; 

» Ten projects linked one or more short-term 
outcome (s) to a long-term outcome; 

» 21 projects linked one or more activities to 
a short- or long-term outcome; 

■ 16 projects displayed more linkages than gaps 
between activities and short-term outcomes; 
and 



Table 1 

ILLUSTRATIVE COMPONENTS OF PROJECT DESIGN 



Types of Input Types of Agtivillfis 

• Staff on boajd • Hire staff 

• Partnersh'q} e^^ished • Davetop acMsory lx>ards 

• Equiptront available • Devetop cumcukim materiate 

• Federal Resource $ • Develop strate^es for recruiiment 

• State Resource $ • Develop dtssemffiatton plans 

• Fittest materials 

• Match tratnif^ to partner requirements 

• identify industry skills 



Types Ql Short-Term Outcomes Types of Lon g-Term Ontfton^ 



• Competency statenmnts • increase (^aHfied iatxv pool 

• Curriculum matenats • Attract new business 

• Accf«dita»x)n sidelines • Creae a competitive high-tech ^ 

busif^ss ami kidj^ry 

• ^udents enroted 

• Grt.i»r equity 

• Students graduate 

• Increase equHy 
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« Five projects displayed more linkages than 
gaps between short-tertn and long-term 
outcomes . 

Figure 3 shows typical linkages among project components. 
However, the most conmon linkages were between: the number of students 
trained and courses offered; skills identified by Industry and related 
curriculum; and number of students trained to Improved labor pool in 
the consnunlty. 

Analysis of the Prolect logic , logical projects met the five 
conditions outlined above. Table 2 displays each project against each 
condition and the total for the project across the five conditions. 
Eight projects received a high logic rati no. ten projects received a 
medium rating, and five projects received a low rating (see Table 3). 
Portland Community College, for example, proposed a highly rated 
project which met all five conditions: 1) identified at least one 
long-terra outcome, 2) linked at least one short- terra outcome to a long- 
term outcome. 3) linked at least one activity to short- and long-term 
outcomes, 4) showed more linkages than gaps between activities and 
short-term outcomes (+14 linkages), and S) showed more linkages than 
gaps between short-term and long-term outcomes (+4 linkages). In 
contrast. Southern Growth Policies Board only displayed a long-term 
outcome and no linkages, meeting only one of the five conditions. 

Overall, the applications tended to identify longer range outcomes 
and to link activities to short-terra outcomes. To demonstrate a 
logical design, however, the applications also needed to identify 
additional linkages. First, they needed to show relationships between 
a short-term outcome and a long-term outcome. Although 21 applications 
linked at least one short-term outcome to long-term outcomes, they did 
not do so consistently. Only five pr- jects consistently linked short- 
terra outcomes to long-term outcomes. 



Figure 3 
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applMfooiS 
fCM^faKiustry 


^ 



I^^emlnate 
program n^xto) 



A A 
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ConcSttons fc^ Pro^ logic 










XIBUl 






Pipjed 


Design must 
loantuy i or mofB 
toi^-term outoomas 


Design kfartttfiT0 1 
nK>fe shoft-term 
oiitcxHnM that are 
Infcedtotong-tenn 


Des^^^^^ftesl 
ormored(^)M 
ft)i(8dtosh(^-(M' 
h)ng4erm outcomes 


Des^nrmMsthave 
nmiatMu^ethan 

i^fvi^andshCKt- 
ternioutoorrm 


Design m^h^ 
imraHfdc^sthan 
g^ b^ween shoit- 
ami (ong^wm 
outcomes 


Total 


1. Divtelon trf Vocalton^ E<kicatton 
Servteas 






- 






4l 


2. Skyttr^ CoHege 






4 


4 
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3. Ventura C<mm^nly CoBdgd 




4 


4 








4. Vatencte Comniintty Cofle^ 




4 


4 


4 


— 
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5. V&^ancta Comrmintty Cotega 
(fiim) 




■ 


4 


4 












4 


4 




+3 


7. Waubonsad Convmmity Cotlaga 




4 


-I- 


A. 




+4 


8. Indian HtSaCommunttyCoVaga 




4 


4 








9. Hampdan County Ernptoymant 
and Trabib>g Consortium 






4 


4 


- 


+3 


10. Centrat Convnun% Ccritega— 
Ratta Cartpi^ 




4 


4 




4 


44 


1 1 . Postsa^ndarv VocMionaU 
Tedinica] Edu^^n 


4 




4 


4 




+3 


12. Southern Growth Polbtos Board 


+ 










+1 


13. University of Noith Dakota- 
Lake Region 






4 


4 




+3 


14, Toledo Ptit>llc Schools j 


^ J 




4 


4 




+3 
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Conctittons for Proidc! LOQic 






Ong 








£bes 






Project 


Dssinn mi si 
ktenMy 1 or mors 

Mrl Iff Slrl 11 1 WWtwVlf 


vt^HMm^ if 11*1 iiio 


or mora m^itt» 

UnKBO HI SJTOn* uf 


uosjgn musi nsvs 
momUnkagastlian 


D8sign rmist have 
morettr^^^s than 
gaps batwean short- 
ami tong-term 
outcomas 


Total 


15. 


France Tuttto Vocitftonal 
Technica} Centar 




4 


4 


4 


4 


45 


16. 


Rutland ConmunHy Coltegs 




4 


4 


4 


4 


45 


17. 


htofthan^on ConrninHy Cot^e 


4 




4 
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IB. 


IfMtena Univsrsity of 
Pennsyfvania 




4 


4 


4 


4 


45 


IS. 


GfeenviRs Techntoal Cotege 






4 
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20. 


Et Paso Comrraintty Cofiege 






4 


4 
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21. 
22. 


Richlaml School District 

Yakima VaJtey Commimfty 
CoSege 


4 


4 


4 

4 , 


4 
4 


4 


45 
43 


23. 


University of Wisconsin— Stout 


4 




4 
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Table 3 

SUMMARY OF PROJECT LOGIC 



in Pfo|oct Logic 


MedhKn 


low m Project Logic 


Meets ait 5 
Coixfitkms 


Meets 4 Condftfois 


Meets 3 ConcMions 


M^s2Concfitions 


Meete 1 CoiKfitbn 


France Ttmid 
Vocatimat Ted^nbai 
Center 

Porttend Community 
CoB^ 

tmilana Untvarsfty of 
Pennsyt^ia 

Rk:htand Schooi Dtetrict 


VenttiraConmmlty 
Cottage Distrtot 

V^nda CorTHmintty 
CoBaga (manufaimirifv) 

Waut>onse8 Conmmfty 
CoH&ga 

Cantral Conmjntty 
CoSaga-Ptatta Cam^ 


Skyib^ Cotega 

Vitonda CwmtfWty 
CoD^ ((ton) 

Pafk^nd Cotia^ 

ImSsm Conminity 
Coflega 

Hampcton County 
En^^cq^neTft and 
Trabh^ ConsiMkmi 

Vocattcmal Technic^ 
Ei^ication 

Toiado Put>tiG SchiK>)$ 

El Paso ComnHJntty 
Cofiaga 

Yakima VaHey 
Comntuntty Coi^e 


Convmmliy Coi^ 
On^anviRa TedirHcal 

IMiv&fStty (A VWsconsln- 
Stoui 


C%ri^ of Vocattonat 
Eduotftonal Saivteas 

&>utham Growth 

- 
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Second, the projects needed to clearly indicate how the resources 
will be used for conduct of the activities. Many projects indicated 
the person{s) to be hired or whose time would be bought. However, they 
seldom indicated what the person would be doing to accornplish the 
outcomes. 
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8. Assessing the Plausibility of Project Design 



This section describes the procedure for assessing the 
plausibility of the project based on the logic model. 

Plausibilit y Defined 

A plausible project is one that has some likelihood of achieving 
its short term objectives. "Short term" means within the 18 months of 
the grant award period. Project objectives may be implausible because 
schedules are unrealistic, resources are insufficient, or because 
available knowledge suggests that the project is not likely to achieve 
its objectives. Assessment plausibility consists of determining the 
degree to which a project is: 

■ Well defined; 

• Can be completed with available resources; 
and 

■ Can be completed within the available time. 

From the logic model, the study team judged the plausibility of 
the set of events and causal links between inputs, activities, and 
outcomes. The outcomes were considered plausible if project activities 
would achieve progress toward the outcomes. For each outcome, the 
following questions were asked to determine the project's plausibility: 
1) are there adequate resources to achieve the outcome?; 2) is the 
schedule for achieving the outcome reasonable?; and 3) do the 
activities suggest an understanding of the necessary steps to achieve 
the outcome? The answers to these questions were recorded in a suimnary 
chart arraying the projects. 

Procedures for Assessing Plausibility 

The study team first reviewed the logic model and the grant 
application to determine whether there were adequate resources (inputs) 
for each proposed activity. Each activity supported by at least one 
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Input was given one point. One point was subtracted for each activity 
not so supported. Second, the study team assessed whether all the 
proposed activities could be performed within the 18 month grant 
period. Each activity that could be completed within the grant period 
was given one point. One point was subtracted for each activity that 
could not be completed. Third, the study team determined whether 
proposed short-term outcomes could be achieved within the 18 month 
grant period. Each short-term outcome that could be achieved within 
the grant period was given one point. One point was subtracted for 
each short-term outcome that could not be achieved. Fourth, the study 
team determined whether there were outcomes (either long-term or short- 
term) for the two required objectives for the Cooperative Demonstration 
Program: 

1. Access to quality vocational education 
training; and 

2. A successful partnership between public and 
private sectors. 

For each long-term or short-term outcome in the project design that 
matched a required outcome or a desired outcome, the design was 
assigned one point. 

Projects were considered "plausible" if they wet all four 
conditions. Projects that did not meet all four conditions were 
considered " less plausible- rather than " not plausible" since it cannot 
be determined absolutely whether the scores are a result of information 
missing from the application or are an actual lack of planning on the 
part of the applicant. 

Results of the Plausibility Assessment 

The projects were scored on four criteria described above and the 
results are presented in Table 4. It was found that: 



Table 4 

RESULTS OF PROJECT PUUSIBILITY 



Project 


Sufficioni 
Resources 


Sufficient Time 


«jiiic»ra Tttne 
for Short-Term 

OutOSfTOS 


^x^ssto 
Trainif^ 




Partnefship 


Total 


UflWlvn Oi VOCmlCliial CiMJCaouri 




































Ventura Communtty CoNege (^strict 
Moc^parK CalHomk 


















Offamto, Ftofida (manUddurtng) 




T 


▼ 




▼ 
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VarvnCia uOmnuiriiiy lh7»99v 


















V/fl9lf^/lil||flY iUliiUKi 


















Waubon^ Cormnunity Cotog^ 
Sugar Grove* Mtinois 
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Indian HiHs Corrumintty Conege 
ORurniva, Iowa 
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Hanmdsn Countv EniDlovman! and 
Train^ Consonium 
SpfiF^iekj, Massachusetts 
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Centra! CormuJntty Cdtege— 
Platte Campus 
Coiumbus, NebrasKa 


♦ 
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Postseconda/y Vocattonal^ 
Technicdi Education 
Concord, New HaiT^Jshtre 














43 





(Continued on next page) 



Table 4, (Continued) 



Prefect 



Sufficient 
Resources 



for Short-Term 
Outcomes 



^x^ssto 
Training 



Public/Private 
Partnersh^ 



Total 



Soufi^m Growth PoHo^ Board 
Ftos^m^ Triangle Park, North 
C^amHna 

Univ^tty d Nottt) Dakota-4.akii 
Regton 

De^^ Lake. North Dakota 

To^ Pubiic Schoote 
Toieck, Ohio 

France Tuttie Vocation al Technk^ 
Center 

Okbhonta City, Okiahcrnia 

Portland Coimiunjty Colte^e 
ft^iand^ Oregon 

ftortharr^n Convmmtty College 
&tl*hem, Pennsyhr5»nJa 

tm^a Universtty os \ ^nnsyiVania 
tndiana, Pennsytvania 

GnronvMe Techntoai Cottage 
Green^tOe, South Carotina 

Ei Paso Conmunity Cottage 
B Paso, Texas 

Richtam} School District 
Kennewk:k, Washbigton 

Yiritlma VaBey Cortimuntty College 
Y^ima, Washbigton 

Universfty of Wisconsin—Stout 
Menomonfe, Wisconsin 



+2 
+3 

43 

44 

44 

♦1 
43 

43 
42 
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■ Eight projects appeared to allocate 
sufficient resources to conduct the 
activities; 

■ 21 projects appeared able to complete the 
proposed activities within the 18-month time 
frame; 

■ 21 projects appeared able to achieve the 
short-term outcomes within the 18-month time 
frame; and 

■ 15 projects met the required objectives of 
the Program: access to training and a 
successful partnership. 

Analysis of the Plausifanitv Assessment . As displayed in Table 5, 
four projects were scored as "highly plausible", twelve were scored as 
"plausible", and seven were scored as "less plausible". 

Projects were rated highly plausible if they allocated sufficient 
resources for conducting the project, proposed activities and short- 
term outcomes that could be achieved in the 18-month time frame, and 
met both required objectives for the Cooperative Demonstration Program. 

Valencia Coranunity College (manufacturing), for example, met all 
four conditions. It proposed to train over 500 students in eighteen 
months. To achieve this, the project focused on recruitment (e.g., 
assigning a staff member to serve as a on-site counselor for students), 
adapting existing course curriculum to meet the needs of students, and 
matching curriculum to Industry requirements. 

In contrast, projects receiving a low plausibility rating met only 
one of the four conditions. Five out of six projects receiving a low 
rating did so because of Insufficient resources to conduct the 
activities proposed, while four out of six projects received a low 
rating because they did not meet both of the required objectives of the 
Program. 

The most common failure of projects was lack of sufficient 
resources to conduct all project activities and achieve the outcomes 
proposed. If this condition is relaxed, at least 14 projects are rated 
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SUMMARY OF PROJECT PLAUSIBILITY 
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as highly plausible rather than only four projects. Although a 
required objective, only sixteen projects were rated as clearly 
intending to develop or maintain a partnership. Twenty-two were 
identified as providing training. 
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L« Sutranarv of Evaluabilitv Assessment 



In this section, 23 FY1988 projects were assessed on the logic and 
plausibility of their project design as stated In their grant appli- 
cations. Project logic was used as an indicator of the "clarity" of 
the project design, while plausibility was used as an Indicator of the 
"coherence." It is assumed that both project logic and plausibility 
are important predictors of project success and the ability to measure 
that success. Most projects received a different ranking on logic than 
they did on plausibility. Table 6 shows the logic and plausibility 
rating for each project, and ranks the project. 

For a project to have a good chance of success, it must be both 
logical and plausibile. A project with a high combined rating will be 
likely to accomplish the outcomes and activities stated in its 
application. Projects receiving a low rating in either plausibility or 
logic are less likely to be successful. Projects receiving mid-range 
ratings in logic, plausibility, or both, are rated as potentially 
successful. Taking both factors into account, three projects were 
predicted to be successful; ten projects were predicted to be 
potentially successful; and ten projects were less likely to be 
successful (see Figure 4). For example, Valencia Community College 
(manufacturing) and Ventura Community College District were both 
predicted to be successful projects from the evaluability assessment. 
Each received a high rating in both logic and plausibility. 

The extent to which logic and plausibility can be used as 
predictors of the success of the project can be tested by comparing the 
rankings developed here with the results of the implementation analysis 
and cost-benefit analysis conducted In the next two sections. The 
relationship of the evaluability assessment to project success is 
discussed In Section V. 
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Table 6 

SUnHARY OF PROJECT LOGIC AND PLAUSIBILITY 



Project 



Project 
Logic 



Project 
Plausibility 



Valencia Community College High High 

Orlando, Florida (manfct) 

Portland Community College High High 

Portland, Oregon 

Ventura Community College District High High 

Moorpark College 
Hoorpark, California 

Waubonsee Community College High Medium 

Sugar Grove, Illinois 

Francis Tuttle Vocational Technical Center High Medium 
Oklahoma City, Oklahoma 

Richland School District High Medium 

Kennewick, Washington 

Central Community College-Platte Campus High Low 

Columbus, Nebraska 

Skylint* College Medium Medium 

San Bruno, California 

Valencia Community College Medium Medium 

Orlando, Florida (film) 

Parkland College Medium Medium 

Champaign, Illinois 

Hampden County Employment and Training Medium Medium 

Consortium 
Springfield, Massachusetts 

Postsecondary Vocational-Technical Education Medium Medium 
Concord, New Hampshire 



Table 6, (Continued) 
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Project Project 
Project Logic Plausibility 

Toledo Public Schools Medium Medium 

Toledo, Ohio 

Yakima Valley Conimunlty Medium Medium 

Yakima, Washington 

Indiana University of Pennsylvania Medium Low 

Indiana, Pennsylvania 

El Paso Community College Medium Low 

El Paso, Texas 

University of North Dakota-Lake Region Medium Low 

Devils Lake, North Dakota 

Indian Hills Community College . Medium Low 

Ottumwa, Iowa 

Northampton Community College Low High 

Bethlehem, Pennsylvania 

Greenville Technical College Low Medium 

Greenville, South Carolina 

Division of Vocational Education Services Low Medium 

State Department of Education 
Montgomery, Alabama 

Southern Growth Policies Board Low Low 

Research Triangle Park, North Carolina 

University of Wisconsin-Stout Low Low 

Mefiomonie, Wisconsin 
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Rgure 4 

PREDICTION FOR PROJECT SUCCESS 



Project Logic 
High 



Medium 



Low 



Project Plausibility 




H » predcted to be $ua:essful 

113 - predicted to be somewhat successful 

Q B predicted to be less successful 
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III. PROJECT IMPLEMENTATION 



fhe second question answered by the evaluation Is to what extent 
did the projects implement the activities they originally proposed? If 
the projects changed their activities, under what conditions were those 
changes made and what were the impli cat ions for project operations and 
accomplishments? Did the activities that were carried out meet project 
objectives as originally developed or as revised? 

This section is organized into six parts. Part A describes the 
procfcwures for selecting sites and collecting data on project 
implementation. The evaluation utilized telephone surveys and site 
visits of a subset of eight of the 23 high technology projects. Part B 
discusses the implementation issues that provided the framework for the 
inquiry. Part C presents an overview of each of the projects surveyed 
and visited, and a discussion of project administration and 
partnerships. Part D discusses the content of the projects and the 
intermediate outcomes from- the activities conducted, including student 
recruitment and assessment, training content, curriculum development, 
skill identification, and staff development. Part E addresses the 
concept of innovation in the projects and notes whether the projects 
endeavored to test new Ideas or simply to replicate already proven 
concepts. Finally, Part F suiwnarlzes additional implementation issues 
r.^quiring further attention In the Year 2 of the study. 



A. Procedures 



To analyze implementation, the study team: 1) refined a set of 
Implementation issues outlined in its technical proposal; 2) selected 
nine sitis for further study through a preliminary "evaluabllity 
assessment;" 3) conducted a telephone survey of the nine grantees; and 
4) conducted site visits to eight of the nine grantees. The team 
conducted these steps as quickly as possible because the evaluation 
contract began in March, 1990, only three months before the FYI988 
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grant periods ended. The survey and site visits had to be complete 
prior to the end of the projects* grant periods in June, 1990— 
insufficient time to use a full-scale evaluability assessment to select 
sites. Instead, sites were selected through a preliminary evaluability 
assessment. First, applications were ranked "high" or "low" according 
to three criteria: 1) the clarity of project objectives and outco^s, 
2) the precision with which project activities were described, and 3) 
the extent vf the project's focus on high technology. Second, the 
applications were divided into two groups based on total rankings. 
Within each group, projects with "low" rankings on high technology were 
eliminated. Finally, projects were selected from both groups so that 
they varied with respect to project focus (training, curriculum 
development, dissemination, etc.) as well as with respect to technology 
fields, the number of activity sites, and the number of participants. 

The final list of nine projects is presented in Table 7. Eight 
projects were included in both the telephone survey and site visits. 
The ninth, Francis Tuttle Vocational Education Center, was included in 
the telephone survey but was designated an alternate site if one of the 
other eight sites became unavailable. The jBore systematic and 
intensive evaluability assessment (described in Section II) arrived at 
different rankings. 

The short time frame necessitated switching from mail to telephone 
surveys. Questions were open-ended and were intended to collect 
information about project startup, changes in levels and types of 
services provided, and changes in expected outcomes. The survey 
obtained preliminary information and was followed up in the site 
visits. It was not, however, used as a pilot test for a survey of 
FY1989 grantees as originally planned. The site visits were conducted 
during May, June, and July, 1990, by members of the study team. 



Table 7 



FY1988 GRANTEES SELECTED FOR THE SURVEY AND SITE VISITS 



1 

— 






cml liaic 




VISIT 


Skyline College 

San Bruno^ Catlfornid 


Toycta/Skyllne Partnership for 
Automotive Technician Training 


Dec. 


May 
1990 


Yes 


Yes 


Ventura torn. College Disl. 
Moorpark College 
Noorpark^ California 


tton-college Bound Student 
DeiDor^tration Project - 
Electronics/laser/ElectrO'C^tics 


1^ 


T990 


Yes 


Yes 


Valencia Coemunity College 
Orlando, Florida 


A Model, Heplicable Advanced 
Hanuf »ctur ir^ 
Demonstration Proj^t 


Jan* 
1^ 


Jttf^e 
1990 


Yes 


Yes 1 


1 Valencia Com^tty College 
1 Orlando, Florida 


Film Proctetlon Techr^logy 
Training Program 


Jan* 


June 
19W 


Yes 


Yes 1 


fr^icis luttle Voca ional 
Technical Center 
OklahOBkB City. Dkl^om 


Hied) TechnotOQV Parlner&hio Prde^t 


1989 


1990 


Vac 

yes 


HO 


Morthanpton Conmmity 
College 

tethlehem^ Penrtsyivania 


Turn-key Surface Kount Training Progr^ 


Jan. 
1989 


Juw 
19W 


Yes 


Yes 


Indian tftiiverslty of 
Pennsylvania 
Indiana^ Fennsytvania 


northwestern Pennsylvania Cooperative 
Demonstration for Technical Updatir^ 


Jan. 
19£9 


June 


Yes 


Yes 


Jlichland School District 
Kennenick, Washingicn 


Materials Technology; The Condon Core 
Skills That ^re Shaping the Fut^^e 


Jan. 
1^9 


Jwe 
1990 


Yes 


Yes 


University of Wiscor^tn- Stout 
MenoiDonle^ Wiscor^fn 


l^lefDonting a High-Tech 
Training Ptoctel for Rural iased Swiness 
md Industry^ Teciinical Colleges^ &ni 
local and $t« Education A^neies 


Jan* 
T989 


Aug. 
19W 


Yes 


Yes 



40 

B. The Framework for the ImpTetnentatlon Study 



Early in the project, the study team identified questions to 
collect information on project administration, operations, intermediate 
outcomes and extent/kinds of innovation. These Issues provided the 
framework for the telephone survey and site visits. They are briefly 
summarized here. 



PROJECT ADMINISTRATION: 

1. Project Infrastructure ; To what extent did the 
project succeed in establishing the 
infrastructure to carry out Its acti.ities? 
Among the specific items that were examined were 
staffing, obtaining needed space, acquiring 
equipment, keeping track of funds, students, and 
other necessary recordkeeping, scheduling 
services, conducting evaluations, and seeking 
additional support (If needed). Did the extent 
to which the project was new to the grantee 
Influence its ability to carry out 
infrastructure development? What other factors 
influenced success in project administration? 

2. Partnership Role : What kinds of partnerships 
were established— i.e., what was the range of 
partner types, what roles did partners play in 
project activities, what was the extent of 
partner Involvement? To what extent did the 
roles played by the partners reflect the range 
of partnership roles identif-^ad in the 
legislation? Were particular kinds of 
partnerships more or less capable of being 
Implemented successfully? 

3. Start-UD Activities : What activities were 
undertaken in preparation for providing direct 
services to students or other clients? How long 
was the start-up phase? Assuming a main 
activity was student training, to what extent 
did projects identify employer needs or 
available job openings, develop student 
assessment techniques, adapt/adopt curricula 
developed prior to the project, obtain 
clearances or other approvals in order to offer 
training, or train Instructional staff? 
Assuming a main activity was curriculum 
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development, to what extent did the project 
identify needed skills, encourage industry or 
other reviews of materials, or pretest 
materials? If projects planned to coordinate a 
set of services, to what extent were other 
service agencies enlisted in service provision 
or other activities mounted? 

4. Student Re cruitment : What activities were 
mounted, who was involved and for what 
percentages of their time? What, if any, 
incentives were there for students to enroll? 

5. Dissemination : As these were demonstration 
projects, what activities were planned to 
disseminate project materials, findings, models, 
etc. and were those activities implemented? 



PROJECT OPERATION: 

Once beyond the start-up, how did the project 
operate? What kinds and amounts of training 
or other direct services were provided to 
students or other project clients? What 
problems were encountered and how were they 
addressed? The evaluation attempted to 
describe the nature of the service and 
develop a means to quantify the amounts of 
service. 



INTERMEDIATE OUTCOMES: 

With respect to training, how many students 
or other clients received services and how 
much service did a typical client receive? 
What were the characteristics of clients with 
respect to common descriptors such as sex, 
race, ethnicity, age, educational background 
and the like? To what extent did students 
complete the training or other service in 
which they were enrolled? With respect to 
curriculum development, what curricula were 
developed and how successful was their 
Implementation? How was information on job 
or skill needs incorporated into course 
development? How were curricula ass'issed? 
With respect to partnerships or other 
linkages, were they formed and what did they 
contribute to the project? 
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INNOVATION; 

Since these projects were intended as 
demonstrations, did they try new approaches 
and, if so, what can be learned from studying 
them? 



These, then, were the questions that guided the inquiry. The next part 
describes what was learned from the survey and site visits. 
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L Implementat ion Findings 

This part reports on Implefnentation findings from the analysis of 
the survey and case studies/ With only eight projects, It is 
difficult to generalize to all high-tech projects supported through the 
Cooperative Denjonstratlon Program. 

L Overview o f the Prelects 

The following paragraphs present a brief overview of the plans and 
activities of each project, noting the original design, the primary 
activities carried out, how they differed from the plan, as well as 
major project outcomes. 

Richland School District! The original plan, largely implemented, 
was to introduce a one-year course In materials science and technology 
(MST) to seven high schools and one coranunity college. The MST course 
curriculum had already been developed by a teacher at Richland High 
School with support and technical assistance from Battel le Northwest 
Laboratories (the private partner). The course was a 180-hour, hands- 
on, science and vocational course about glass, ceramics, metals, 
composites, and wood. Two teachers from each of the seven high schools 
were trained in a three and a half week summer workshop. They adapted 
the MST outline and workbcyok to their own school and local needs, and 
trained 237 students during the 1989-90 school year. One site dropped 
out at the mid-year point because of school construction and staff 
turnover. 

Northampto n Conmiunitv College : The National Training Center for 
Microelectronics at NCC proposed to provide local manufacturing 
companies (the private partners) with customized job training in 
surface mount technology (SMT). During the grant period, the Center 
expanded its existing training program in SMT, enhancing four existing 
courses, creating seven new courses, acquiring new equipment, and 



■ * As appropriate, the report also references information from the Francis 

Tuttle Vo-Tech Center collected during the telephone survey. 
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producing a national teleconference. The project trained 233 employees 
of seven aiicroelectronlcs firms during the grant period at reduced or 
no cost to the companies. The project also reached an estimated 2000 
employees at 18 locations nationwide through two four-hour 
teleconferences, one of which was interactive. The subject of the 
conferences was "Packaging In the 1990s," and videotapes of the 
teleconference were sold to private companies and donated to other 
universities. The project was generally implemented as proposed, but 
trained only about half as many students as planned because companies 
enrolled fewer students than predicted. 

Valencia Communltv College [filml : The VCC staff established as 
proposed a film production technology program to train students for 
jobs In the growing local film industry. With technical support from 
Universal Studios (the private partner), VCC developed the program's 
curriculum and implemented a 15-week course offered three times during 
the grant. A total of 135 students were trained in stagecraft, sound, 
set construction, camera/editing, and post production. Together, the 
three sessions produced a full-length feature film entitled "Sealed 
With A Kiss." The project was Implemented as outlined in its 
application. 

Valencia Communltv College rmfg.l : VCC proposed providing a local 
manufacturing company with customized job training in automated 
manufacturing technology. VCC was already working with Stroroberg- 
Carlson, Inc. (the private partner) under a State of Florida grant to 
help Strofflberg- irl son introduce high technology-based manufacturing 
processes. Oui ng the Cooperative Demonstration Program grant, VCC 
trained 565 Stromberg-Carlson employees with the curriculum designed 
under the earlier grant. A total of 26 classes were offered in 17 
different courses; the courses averaged 65 student contact hours. The 
project was implemented as proposed, although 30 percent of the 
students dropped out prior to graduation due to other demands on their 
time. 

University of Wisconsin-Stout : The project proposed implementing 
a "model" high technology training program in three technical colleges 
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and 12 high schools in University of Wisconsin-Stout's service area. 
Project staff conducted an initial four-week summer workshop and 
trained teachers from the participating secondary schools and 
postsecondary technical institutes how to conduct local needs analyses 
and to develop module curriculum. Teachers at each school interviewed 
local industry officials to determine high-tech training and skill 
needs and to develop course modules. Teachers spent the 1989-90 school 
year preparing their BJ0dule{s) and testing them In classes. The 
project planned to coaplete 41 modules— ranging from a few hours to a 
full semester of instruction. The project was implemented as proposed 
except that teachers at the participating technical colleges did not 
have time to begin training employees of the private sector partners. 

Skyline Community ColleQe! The vocational division at the college 
proposed a joint training program for service technicians with Toyota 
Motors Sales (the private partner). The program, called T-TEN, 
included 16 weeks of formal instruction per year for three years and 
part-time work at a reduced wage in Toyota dealerships. Toyota 
provides financial incentives to the college and to students for 
implementing and graduating from the program. The project was 
underway, with state financing, prior to the Cooperative Demonstration 
Program grant. Potential students were recruited through newspaper ads 
and selected by dealership personnel during an annual meeting. The 
College began the process of obtaining national certification (NATEF) 
for its automotive program. At the end of two years, a total of 17 
students were enrolled in the program and two Skyline instructors had 
received extensive Toyota training and developed the training 
curriculum using Toyota materials. The project was largely Implemented 
as planned, except 1) there were fewer participants because there were 
fewer job opportun1<-1es with Toyota dealerships, and 2) most students 
ended up working full-time because they needed the money and dealers 
needed the staff. 

Hoorpark Conrniunitv College : The original plan was to establish a 
coordinated high school -col lege program in electronics and 
laser/el ectro-opt^cs for at risk students in eight high schools. In 
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addition, the local business/ labor council (BLC— the private partner) 
would arrange for field trips to potential employers, transportation 
among sites, counseling, and other activities. The project curriculum 
was to be developed at the college. The initial plan was modified 
considerably over the grant period: 1) a sunroer remedial basic skills 
program was not held, 2) high school instructors received informal 
training from the college as needed to implement the program, 3) 
planned inter-school transportation was simplified, and 4) the BLC role 
diminished substantially. As jobs in laser-optics decreased, college 
attendance became a more likely student outcome. In the end, however, 
the schools adopted the college-developed program. Fifty- two students 
from eight high schools completed the coordinated instructional 
program, and more are now enrolled in the College. Students received 
instruction at four of the schools for four days a week and at Moorpark 
one day a week. 

Indiana University : The initial goals of this project were 
ambitious— to establish collaborative technical training between 
Indiana University, county vocational schools, other postsecondary 
institutions, private trade schools, regional economic development 
agencies, and the private sector. Problems in coordinating with the 
regional economic development agency, however, scaled back proj<}Ct 
goals. In the end, the project delivered a variety of short-term 
training programs to the employees of smaller manufacturing and other 
firms in the area (the private partners) at no cost to the companies or 
employees. Courses varied from basic math to the use of sophisticated 
computer controlled machinery. Most courses were offered by the county 
vocational schools and approximately 648 students received training or 
attended product demonstrations. Thirty-three classes were organized. 

Francis Tuttle Vocational-Technical Center: (telephone survey 
only): The goal of this project was to increase enrollment in the High 
Tech Center within the institution by providing academic remediation to 
adults who would not otherwise qualify, and to extend formal 
instruction with internships. This project was largely implemented as 
planned. A recruitment campaign was undertaken and a self-paced 
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learning lab installed. To attract students, the project provided 
tuition reimbursement for 220 students without regard to financial 
need. Eighty percent of the Center's students used the lab. 
Ipf^ernships with stipends paid from the grant were provided to 20 
students (sites were the private partners). The project recorded a 
substantial increase in enrollfflents and a dropout rate of 30 percent. 

2. Project Administration 

Timing of Award; Applicants suddenly awarded grants may suffer 
timing problems. In the first year of the Program, application 
Bjaterials mentioned an October 1988 award date, but applicants were not 
notified of awards until October 1988 and began their projects at their 
convenience after that. All but one visited projects began spending 
funds in January 1989, but most indicated that the timing of award 
notification was difficult. Most grantees were academic institutions 
operating on a 9-roonth academic year that had begun in September. By 
the time the grants started, almost half their potential "planning" 
year was over. Notification in the fall would have allowed an eight 
month period in which all school personnel were readily available. 
Furthermore, had the grant award started early in the academic year, 
project activities could have been incorporated into the overall 
planning activities of the grantee organization. 

Projects that were already involved in the activities for which 
Federal support was sought were better able to acconmiodata the mid-year 
start date. Three projects (Richland, Valencia [film] and Valencia 
[fflfg.]) were already training students or had undertaken extensive 
planning before the grant began. Three other projects (University of 
Wisconsin-Stout, Skyline, and Northampton) were actively planning 
before the grant started although some of these projects still 
encountered start-up difficulties. 

The grantees most affected by lack of time were those beginning 
new enterprises with the awards. At Moorpark, for example, most 
curriculum development and teacher preparation activities were carried 
out during the Summer of 1989 and many adjustments in operations were 
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made after students were already attending classes. When problems 
arose In the Spring of 1989 with the partnership arrangaaent at Indiana 
University, there was not enough time to rethink the design before 
actual training begun. 

Hiring Staff : Most projects did not plan +0 hire new staff in key 
positions; of those that planned to, few succeeded. Project 
administrators blamed the timing of awards, and the 18-roonth time 
frame. Recruiting and hiring took several months, and some positions 
went unfilled. Grantees had to prepare job descriptions, post notices, 
interview candidates, and negotiate the terms of employment. In two 
projects, people had been provisionally recruited to run the projects 
in the Fall of 1988, but by the time the grants started and the 
institution's hiring procedures had been met, the individuals had taken 
other jobs. These projects decided not to fill the position with new 
hires either because no qualified candidate would take the position for 
only 18 months (or less by the time the recruitment procedures were 
fulfilled) or because the grantee did not want to incur the long-term 
responsibility for employment after the grant ended. In both cases, 
project director responsibilities were divided among existing staff. 

Several projects shifted responsibilities back and forth among 
staff members over the project period and had not achieved a staffing 
"groove" until the grant was ending. This was most apparent In 
projects that sought new hires, but was also a problem in projects In 
which In-house personnel had taken new jobs or other staffing changes 
had been made between submitting applications and beginning the grants. 
Several projects had 20 or more people working on the grant-funded 
activity part-time or by-the-course basis, unlikely to facilitate the 
continuation of the project after the grant period. Institutions 
accorraiodate, "soft money" grants inv.o their operations by ensuring it is 
spent on personnel for whom no lone,- term conroitroent exists. This 
approach mitigates against project continuation when the grant ends. 

Purchasing Eouipment : Most projects perceived that grants could 
not be used to purchase equipment, although there was no outright 
restriction on such expenditures in the first year grant awards. This 

7f; 



49 

probably reduced such purchases. Several projects, however, spent 
staff effort to find other ways to fund equipment. One project shifted 
another grant to equipment support when the Federal funds were awarded, 
but others used what they had sought donations from employers, or did 
without. 

Establishing Private Pa rtnerships ! "Partner" had different 
meanings among projects. In most cases, the "partner" was a private 
employer enlisted by the public grantee. In one case, the private 
partner was a nonprofit consortium of businesses and labor 
organizations. In another case, the private partner was a nonprofit 
research organization that developed a curriculum in conjunction with a 
school district. One project had no real partnership with the private 
sector other than having teachers interview local businesspersons. 

Most private partners played limited and conventional roles such 
as offering one-time or periodic advice on curriculum development, 
serving project advisory consnittees, identifying marketable skills, or 
aonating equipment and materials from local businesses (sometimes In 
exchange for training employees). In a few cases, businesses were 
identified as partners If they might be willing to hire project 
graduates (Table 8). 

Three of the eight visited projects were customized training 
programs for employees of "partner" companies (Valencia [mfg.], 
Northampton Community College, and Indiana University); the University 
of Wisconsin-Stout planned to provide such training but ran out of 
time. Customized training programs enlisted employers (partners), and 
"sold" them low or no cost training services, often with the hope that 
the partner would pay for comparable services after the grant period. 
Two projects provided broader training programs— Skyline Community 
College for persons seeking employment in Toyota auto dealerships and 
Valencia [film] for people wanting jobs in Universal Studios, but 
neither guaranteed employment to graduate. Two projects— Moorpark 
Community College and Richland School District offered one-year 
Introductory programs for high school students in particular high 
technology fields but not for specific employers. 
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Table 8 
PARTNERSHIP DEVELOPMENT 



1 IHtaiECT 


RROPOSED 


ACOMPLISHEO 


MOT Acxxmisiieo 




ll« Wisconsin 


& p<irtni^r<4)ips witli secon- 
dary and votech schools 


o cooperative ^reefomts 
with 16 secondary and post 
^ondary schools 

o local partnerships w/bi»* 




0 hi^ tech training 
model included partrar- 
ships 


Hoorpark 


o Private Industry Cm^tl 
|MH>vlda8 training funds 

o other companies purchase 
trainir^ 


o cooperative agreement with 
^iness md lal>or Council 

o cooperative i^reetnent with 
Ventura high schools 




0 other caqpanles donated 
SID^WM) In ec^ipnent 


Skyi tne 


0 be identified with «ajor 

enpt oyer 
o mlist dealerships 
o tUkltf certification 


o cooperatf*^; agree^^nt with 
Toyota 

o enlisted 11 (tealershif^ 
o NA7EF certiflcatic^ 


0 11 dealers not enlisted 


o Toyota excluded cer- 
tain dealerships 


Val&icid Cfifs-I 


o partnership with Stro^ber^ 
Carlson 


o new partn^i^ip with 

St r ooberg ' Car t son 
o other coi^panies ct^trib- 

uted hardware & software 




o existing Stronberg 
Carlson staff 
training program 


Vaiencid liiimi 


o partnership utth 
Universal Studios 


o r;ew partnership with 

i^iversal Studios 
D US alloi^ access to Its 

vendors 




0 previous working 
relationship with US 

o vendors sold equlpMnt 
at reduced f^lces 




V solicit vCw^/oni VrO TvT 

trainins 
0 teleconference with 

indi^try, cotleses^ dssoc» 


o renews parinersnips wicn 
previous corporate cllmts 

0 partnered with teieconf* 
grc&i^ 






Indiana Univ. 


o develop and outreach 

program 
o 22 partners 


0 exlsttr^ mmdBte to serve 
area maticnat schools 

o offered training chei^r 
th^ (rivato schools 


o partnership with ecoooiDic 
development :-*ency felled 

o tittle recrultnmt of 
f in» 


0 private Indi^try 
visited demos » were 
provided training 


Rlchtdnd 


o create new partnerships 
with secondary and post* 
secondary schools 


0 continued existing part- 
nership with Battel le 

o new ^eements with 7 
districts & 1 coom. col. 


0 1 school dropped out mid- 
year due to c<M>$truction 
staff twnover 


o Battelte paid for 

curric* ch^li^jDsnt 
o schools contrlb. $ 
o CIJU helps cufrlcutum 
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The private sector Initiated two projects. The Skyline Collpge 
program was originally developed by Toyota, which had approached 
Skyline with the Idea. The college was one of 50 sites nationwide. 
The Toyota program is representative of partnerships undertaken by both 
U.S. and Japanese car iMdnufacturers to Improve the training of service 
personnel. In Richland School District, the private sector partner, 
Battel! e Northwest Laboratories, had sponsored curriculum development 
as part of a U.S. Department of Energy contract. 

Partnerships roles that emerged from the sites in the case studies 
included: 

■ Partner as customer (customized training); 

■ Partner as advisor board of directors— 
sometimes consulted on curriculum, but not 
always; 

■ Partner as supplier of resources— such as 
equipment; 

■ Partner as instructor; and 

■ Partner as initiator of the relationship or 
project. 

These categories are not exclusive. 

Other PartnershlDS - Some projects enhanced "partnerships" among 
educational Institutions. Joint arrangements ! ^tween secondary school 
districts and postsecondary Institutions were common. For example, 
Moorpark Conanunity College established a relationship with four school 
districts to implement an integrated high school laser-optics 
curriculum. The University of Wisconsin-Stout trained high school 
teachers and worked with them designing and testing curriculum modules. 
Indian^ University tried to strengthen the capacity of area vocational 
institutions to serve private employers. For some teacher training 
institutions these "partnor ships" are part of ongoing programs; for 
others, such as Moorpark, they were the result of the grant award. 
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Other St ' t-up Activities : At the outset, the study team 
identified several activities likely to occur during the project start- 
up, including recruiting students, adapting curricula, establishing 
management information systems, obtaining necessary approvals for 
course offerings, identifying jobs, and conducting public relations 
activities. 

Some projects engaged in elaborate student recruitment efforts, 
depending on -ihom the projects planned to attract. Projects that 
planned to recruit new students from less-advantaged populations 
invested more heavily in recruitment. Those that planned to train 
current employees of a single "partner" or students already enrolled in 
the grantee institution made less effort to recruit. One customized 
training program recruited employers who were interested in training 
employees and then spent time selling the idea to the workers. The two 
projects that offered training that m^ght lead to employment in a 
particular industry (Skyline and Valencia [film]) appear to have spent 
the greatest time on recruitment. Table 9 shows the major 
accomplishments oi the eight projects with respect to recruiting 
students for training by either the project staff or the partner 
organization(s) , 

The other start-up activities were less frequent. Few projects 
introduced management Information systems; most either used information 
systems already in place or did not collect systematic information. 
Obtaining reliable information on numbers of students served, amounts 
of service, and compiling financial data were difficult. 

One project encountered start-up problems when a private partner 
could not provide promised placements for enrol lees. Another project 
found that its plan to gain access to private employers through an 
economic development agency no longer viable and new links needed to 
establish. 



Table 9 
STUDENT RECRUITMENT 



1 raojfCT 






1^1 M»3im.lSH£0 




1 U» Wisconsin 


0 iefl to participating 
schools 


o recruited 45 if^tructc^ 
o voluntary registration for 

classes 
o recnitted 450 stud^ts 






1 Moorpfirk 


o produce press role^es 
o present speeches to poten- 
tial students 
0 spor^or k#orkshop5 


o recruited d& students 
o cour^elors ^si^^ed 
students to class 




0 each school hmdled own | 
recruits»^t | 




o recruit 20 students fo^ 
each of three sessioits 
o develop brochi^es 


o recruited 200 students 
o received SO applicatior^ 
after initial orientation 




0 Med newsp^r acte 
o 12 studi^ts hired by 
Toyota 


Valencia Cmfg.i 


o recruit 461^ students 


o recruited 565 students 
o students were ^toyees 

assigned to training by 

supervisor 






Vaiencia [fUm) 


o prepare and circulate 

fimouncef^ts of program 
o recruit 120 students 


o sent out brochures to 

fsedia, uniorm. Industry 
o 3024 applications 
o accepted 163 students 




o re<^ired students to 
have basic skills 


ttorthanpton 


0 recruit companies for 

trainins 
o recruit '«B0 stuc^ts 


o contacted local industry 
o recruited 233 students 
o students were egployees 
assi^ied to training 


o compmies recruited fewer 
than 15 stud^ts per 
course 


o 2000 students watched | 
teleconfer^xe 0 


Indiana l^iv. 


o recruit f irins for 
training 


o students already e^toyees 
of coi^pMiiesi assi^mf to 
training by supervisor 

o ATVS recrtttted f inas 


o recruited fewer firm 


o partner to do 
recruiting but failed 


1 Richland 


o recruit 250 students 


o distributed brochures 
o recruited 237 students 
o voluntaiv registration for 
Classes 


o 13 students short of goal 


o each school wm 
respcmible for 
recruitinent 

^ 
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D. Project Content an d Intermediate Outcomes 

Projects emphasize one or more of four different activities, each 
with different outcomes: 1) technical training with the aim of job 
placeaient or upgrading, 2) development of curricula to meet identified 
skills needs, 3) testing and dissemination of previously-developed 
curricula, and 4) inservice training aimed at developing new courses. 

Direct training services to students : All visited projects 
emphasized vocational or technical training, but the amount and 
duration of training varied. For example, Skyline Community College 
provided each of Its students with 16 weeks of full-time automotive 
mechanics training per year for two years and planned to offer a third 
year for students to complete the program. Francis Tuttle provided 
more than 2,000 hours of training over 24 months. On the other hand, 
some of Northampton's training modules in surface mount technology 
provided students with only a few days of training. In some projects 
with extensive training— such as Francis Tuttle and Moorpark— Federal 
grant funds were supplemented by the grantee -Institution or the partner 
organization. 

The role of training varied. In two cases, training was provided 
to test new curr'cula. For example* a goal at Richland was to 
disseminate a new curriculum in materials technology, whereas at 
Northampton training was used primarily to test and refine curricula. 
These differences In goals and intentions make cross-site comparisons 
of amounts and duration of training in relation to resources difficult. 
If not questionable. 

Respondents were asked what students might expect if they 
completed training. The most common "reward" was a vocational 
certificate (Richland, Skyline (certificate of journeyman status), 
Indiana University, and Francis Tuttle). Francis Tuttle Indicated it 
was establishing a relationship with a community college to allow 
graduates to receive an associate degree. Moorpark and Northampton 
Indicated that students would obtain college or continuing education 
credits. 
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Grantee applications indicated that programs developed under the 
grant would guarantee completers new jobs or promotions; Moorpark, 
Richland, Valencia [film], Skyline, and Francis Tuttle said that 
students would be likely to gain such benefits, Richland (the students 
at Columbia Basin Community College), Valencia [mfg.], Northampton, 
University of Wisconsin-Stout, and Indiana University indicated 
promotion or salary review was likely. 

Most projects provided short-term training separate from regular 
offerings, although a few tried to develop new components to ongoing 
programs. The emphasis on customized training is itself an indication 
of the short-term focus, and such add-ons tend to disappear when the 
funds end. Table 10 shows the major training activities and 
acccmplishroents of the eight projects. In Year 2 of the study, the 
evaluation will explore how the grant program design framework 
contributes to project design. 

Curriculum Development : All but one of the projects engaged in 
formal curriculum development and at four projects this was a major 
focus of the grant. Northampton developed 11 new training modules and 
"polished" several others; Richland is developing a curriculum 
guidebook to disseminate; and Moorpark developed a curriculum for 
simultaneous use in high schools and the college, and modified the 
curriculum as a result of teacher feedback. Table 11 shows the major 
curriculum development activities and accomplishments of the eight 
projects. 

Despite heavy emphasis on curriculum development, projects sought 
systematic teacher feedback. Almost none of the. projects reviewed the 
curriculum prior to its introduction and none established a means of 
determining effectiveness in the classroom. One project simply 
packaged and distributed the "modules" developed by teachers who 
participated in an inservice training workshop. The granting agency 
should insist on more vigorous curriculum evaluation for curriculum 
development projects funded in the future. 

Staff Development: Few of the projects emphasized staff 
development— probably because grants were demonstrations (staff 
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Table 10 
STUDENT TRAIK.:^G 



WOTECT 








COMMENTS 1 


1 U. Wisconsin 


0 train ^ employees 
o orient 1000 students 
o orimt 300 post -secondary 
stucients 


o 4S0 students trained 

o 10J2 hours of instruction 


o SO oi^loyees not trained 




Moorpark 


o offer remedial Instruction 
o offer introductory 
courses 


o 3S stutents trained 

o 23^936 hi»rs of training 


0 3 stuctents dropped out 




1 Skyline 


o offer Instruction for four 

sessions 
o coordinate ii»ork eitpef ience 

session 


o 1^ students trained 
o 21,760 hours of tralnir^ 
o 2W students trained in 
}Qi> hunting skills 


0 51 students drc^^ped out 


Q no fM^ise of Jobs 
0 %\i mXe 

o {problems with schec^le 
& dealer recruitment 


1 Valencia tafsJ 


0 train 465 stutents 
o offer 18 coui ses 


o 392 students trained 
o 26 cldSses/17 courses 
offered 


o 173 students dropped out 


0 19£ Qinorlty 
o 49% female 


i Valencia iiiitBi 


o train 120 stutenti 


o 135 stunts trained 

o 756 hours of training each 


o 28 students dropped out 


o three sesstor^ of | 
three months each | 


1 Horthdi^on 


0 train 460 students 


o 233 stud^ts trained 
o ^,136 hours trainir^ 
o 2(W trained by telecon- 
ference 


local training 


o f 4^ik Of scuucnts rateo 
cotrses excellent 

0 tel^onference invoi* 
ved 32 sites 


Indiana Univ. 


o provide sk i 1 1 updat ing 
work^iops 


o 648 students trair^ 
o 12^^ hours training 
o 33 courses offered 




o end-of^coi^e survey 
shwed oost students 
satisfied 1 


1 ftichland 


o train 2SC students 


o 237 stud^ts trained C217 

high school, 20 college) 
o 42f6M hours of trsining 


0 13* not trained 


o school reoodeling 
turnover stopped class 

o actoinistrators posi- 
tive about cotrse 



Table U 
CURRICULUM DEVELOPMENT 











CONME«TS i 




0 develop o^rlcutiA ffiater* 
fal8 for elfifi^secondary 
and post^secondary 
achoois 


o 24 new courae moc&iles 
developed 


o 10 new course isoc&iles 

pending 
o 5 cotH^se siodules not 

co^leted 


o time conflicts on t^ | 
port of instructiH* | 


Noorpfirk 


0 none 


o ffiodified existing course 
curriculum 




o developed coc^inated 
ct^rlculum beti^m 
colt^ and 4 high 
schools 




o ffnaUze currlcuiua 
o prepare cc^rse i>^*^?riala 
o develop learning 
objectives 


o devel<^wd nen tests and 
tabsheets for course 

o SIM) manhoi^ across 7 
courses 




o use Toyota manuals as 
texts 




o de$i^ training cirrSci Jk 


o development of sonie new 
c»terials 




o accost ished prior to i 
COO^ grmt using S froai | 
state grant | 

0 itsed stmdard textbooks | 


ValetKid CfUflQ 


o develop fiim production 

l^ograiB 
o de${^ and develop 

curriculum 


o developed new curriculua 
in five fila areas 




o MCoi^Udied prior to 
COOP grmt Msif^ $ froo 
state grant 

o visited other projects 


ttorthaofiton 


o m»kQ site specific s»di^ 
fications to training 
modules 


o 7 new courses develq^ 
0 4 courses upgraded 




o 2 cota-ses ap^ved for 
college credit as part 
of electronics prcgrm 


1 Indiana U^iv. 


0 supply instructors with 
^Impropriate teachif^ 
materials 


0 sow modification of 
cotrse for coop<tegio 
students 




0 used existing curricula 
o student ImMledge and 

^flities varied 

tildeiy 




0 tailoring of outline to 
local labor fliarket 


o f^-flialized previously 
i^vetoped curriculum into 
Ktf)dbook mid experiments 




o short time frame 
o college ^9provat 
IN-ocesa difficult 



85 



58 

development tends to be an ongoing activity with an Inherently local 
focus). One project was primarily a staff developa^nt project, 
bringing together secondary and postsecondary Instructors and helping 
them learn how to construct training modules in new fields. This 
project was effectively "testing" a model in which teachers develop 
curricula through Information provided in workshops. One project 
disseminated curriculum by training teachers in its use. 

Nonetheless, most projects offered some type of staff training, 
although the amounts of varied. Three projects that introduced new 
curricula (Richland, Skyline, and Northampton) provided intensive 
teacher training. Francis Tuttle trained staff to assess students* 
basic skills and to teach reading and math skills in a learning lab. 
Valencia [film] and Valencia [mfg.] both indicated that some staff 
development had taken place prior to the Federal grant, Moorpark 
conducted no formal staff development, but there was informal teacher 
training and teacher-counselor interaction, including a bimonthly 
meeting. Indiana University offered no staff development directly, 
alt'iough instructors spent their own time in curriculum development. 
Table 12 shows the major activities and accomplishments of the eight 
projects with respect to staff development or staff training. 

Curriculum or Other Dissemination : All projects had plans to 
disseminate their findings, but only one made dissemination a major 
emphasis, Richland's was primarily a dissemination project: the 
Materials Science Technologies course was already operating In the 
Richland schcol district and the project sought to disseminate it more 
widely. The grant was used to replicate the course and adapt it to the 
needs of educators and local labor markets. Other project directors 
made presentations at regional (or national) vocational education 
conferences and spread information to community colleges via 
association meetings and conferences. Northampton's project director 
gave lectures and wrote articles for trade journals. Three of the 
projects (Valencia [film], Valencia [mfg.], and University of 
Wisconsin-Stout) Indicated that the Federal grant allowed them 
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Table 12 
STAFF DEVELOPHENT/TRAINING 



1 FftOIECT 






MOT ACCOKPliSNEO 


comENTS 1 




o In-service training for 50 
and post seconciary 
tethers 


o 50 instn^tora trained 
0 40 **04irs of training each 






1 Noorperk 


o none 


o 3 instructors trairi&i 

0 120 hours of trainir^ each 




o exteraive assistance 
to psrticipatfi^ 
high school instructors 1 


Skyline 


o if^tructora participate in 

Toyota adv^^Ked training 
o assi9<^ Inatfuctora 


0 7 staff trained, 2 Inten- 
sively 
o 200 hours of training 






Valencia ImfgA 


p none 


0 nme 




o accompii^^ prior to 
COOP urant tslng S from 
state grant 




o none 


0 n^te 






U Northai^tQn 


0 none 


o seminar lor 94T trainers 







Indiana Univ. 

1 


o none 


0 none 






1 Richland 


j> prov**'^ teacher training 


o 15 instructcH^s trained 
0 160 ho^s of trainir^ each 




o orientation session 
o 3,5 week summer 

workshop 
o 2 1*1^ follow-ups 



ERIC 
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to help other educational institutions replicate their program. Table 
13 snows the major anfamplishments of the eight projects with respect 
to the dissemination of the design, content, instructional materials or 
knowlsdgfc of the project to other vocational education institutions. 

Skills kientificatlon: In their applications, a number of 
projects planned to identify vocational and/or acaderaic skiTIs needed 
in particular occupations, usually where qualified workers were in 
sho»t supply. Based upon the skills identified, they planned 
curnculuni development, assessments of individual skills (prior to or 
after training), and studant recruitment and training. 

In fact, few projpcts achieved these goals. Business 
repres«:ntatives on advisory councils were often consulted about what 
skills they thought were needed, local employers who were not project 
partners were contacted to find out what sort of training they 
demanded. Of the nine projects, one project surveyed local businesses 
and industries, and another conducted a literature review to identify 
skills. Table 14 ihows the major activities and accomplishments of the 
eight projects with respect to the Identification of high technology or 
other skills that students should learn through the project. 

Other oroi ect activities : Seven projects (Moorpark, Valencia 
[film], Valencia [mfg.], University of Wisconsin-Stout, Skyline, 
Northampton, and Francis Tuttle) indicated that they carried out 
assessments of student abilities. Two projects (Valencia [film], 
Valencia [mfg.]) had ieveloped the procedures for conducting the 
student assessments before receiving the Federal grant, while Skyline, 
Univc'sity of Wisconsin-Stout, and Francis Tuttle began assessment 
activities after receiving the grant. Francis TuttU, Valencia [film], 
and Moorpark administered pre-tests as part of their student -.sessment 
activity. Francis Tuttle indicated that students entering the 
institution are usually assessed, but not as extensively. Table 15 
shows the major activities and accomplishments of the eight projects 
with respect to the assessment of students either before or after the 
training was provided. 



Table 13 
OISSEHINATION 



1 FfrOJfCT 


fitOPOSED 


ACOMPliSHEi) 


WOT ACQW^LISHCD 




U. Wisconsin 


o conc^t 5 diss«6iination 

workshops 
Q |H^e«mtatian At state istg. 
o publish project newfttetter 


o (N-esentatiora at national^ 
regional* and state con^ 
ferences 

o project newsletter 






1 Hoorperk 


o none 


o presentation at state 
AACJC 

o presentation at state 
cc^ference 






1 Skyline 


o develop brochu. •s. fliers. 

advert iseraents 
0 prepare articles for 
ind^try newsletters 


D feasant attorn at AACJC 
o presentation at state 
conference 




o iKk^essed promises and 
pitfalls of corporate 

sponsors 


Valencia Imig^} 


o none 


0 presentations at national 
and state confarenctrs 

0 staff Involved in state 
committees 






1 Valencia Ifilm} 


o ncne 


o produced video tape for 
national distribution 

0 submitted i&aterfals to 
ERIC 




o pursuing possible 
coimrcial distribution 


1 Northampton 


o f^oiaote interactive tele*' 

conference 
o participate in state mt9S> 
o visit Pa. coBsx&inity cott* 


o marketed courses to other 

cos|>mles 
o confuted national tele- 

conference to 35 sites 




o second teleconference 
not interactive 


lnd$«\a IMiv* 


o promote training via net^- 
tetters, l^ochures, etc. 


o presentation at AVA 
o f^esentatim at Penn. 
vocatio^l education coni. 




0 poor tumour due to 
logistical setup 


tichtand 


o prepare descriptive mat- 
erials 
t make presentations 
o Viixoit to disseffi. net^rk 


o presentation at AVA 

o presentatitm at state mtg. 

o 2 masazlr^ articles 




o sufimer worksh(^ after 

end of ^r»it 
0 possible publishing of 

notebook as uxtbook 
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Table 14 
Si<ILLS IDENTIFICATION 









NOT ^MXCMPIISMEO 






o conduct hi^- tech cooope- 

teocy swvey 
o identify new technoiodicat 

developoients 


0 ccnckicted ^ Interviews 
with p-ivate indimtry 
0 st^veyed 2SS local fines 
o revieii^ tlteratire 






1 Hoorpark 


0 none 


0 dt>tained skill estimates 
from two local firms 








o establish adwisory 
committee 


o worked with local dealers 
to identify skills 






Valencia {mfg*} 


o assess tralnind neecte of 
industry 


o none 




0 acco^lished prior to | 


1 Vai«fKia Cfflsd 


o assess training neecte of 
IMiversai Studios 


o none 




0 accoaplished prior to 
CQC^ grant usins $ from 
state grmt 




0 analyze cotapany for mod- 
ule cievelopment 


o done in cooperation with 
coopwiies reqifesting the 
training 




o electronic solcferfng 
0 electronic manufac* 
turing processes 


Indiana Univ. 


o perform inventory of skill 
development 


o surveyed 4000 sainufac- 
turers 

0 follow^i^ survey of 30 
local firms 




o <^lity control 
o cu«iputer controlled 
machinery 1 




o to be done by local 
advisory comittee 


o dor^ by local ach^isory 
comsttee 




0 course had already | 
ic^tlfied skills, need 
local modification 
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Table 15 
STUDENT ASSESSMENT 



1 raojEcr 






HOT ACC^)KPllSHE0 


CCMIttftTS 1 


Wlsccn&in 


o none 


o indfvit^t Instructors 
adulnistei' pre and post 
tests 






ffoorperk 


o gxxiify current tests 
o test and interpret scores 
o assess |»^- reading, £n9* 
Hsh, and iiiath skills 


o pre and pcmt- tests 
o special tests of mechan- 
ical skills 








o conduct pretests 
o evaluate progress toward 
learnino objectf^^ 


o tests of Ef^Hsh and math 

skills at orientation 
0 on the job assessioent 




o skill levels of stu- 
dents varied widely 


Valencia CmfgJ 


0 assess training needs of 
2S0 to 46S .j^ptoyees 


o instructors ddninister 

post tests 
o interests and neecte survey 
of SC e^loyees 




0 accoe|>llshed f^^ior to 1 
COOP srant wing S | 
f roe state grant | 


Valencia IfHm) 


0 review credentials of 

fila cffipioyees 
o assess skills of noo-fiim 

Individuals 


o pre and post tests 


n 29 students received 
pre test only 


o approximately 30K i 
Inprovanent in scores | 


Northampton 


o none 


o rone 






Indiana Univ. 


o none 


0 at discretion of Individ- 
iL»i instructor 




0 enpto^ givm option 
of t^ker assessoent 
but none did It | 


Richland 


0 baseline end-of -course 
survey 


o 1 baseline si^vey and 3 

end* of course surveys 
o course exaos ^>d grades 




0 40% students rated | 
course excellent fl 
0 faculty pleased | 



91 



64 



Initial plans showed some projects intended to 1) assess students' 
skills as they begun the program to establish comprehensive individual 
student records, and 2) follow up corapleters. The study hoped to 
obtain exit achievement scores, subsequent education activities, and 
subsequent employment experiences of the graduates from projects with 
good student records and complete training cycles. Aside from overall 
numbers, however, few projects could supply information on the 
characteristics of students served by the grant. 
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L Innovation In Projects 



The purpose of a dewonstration can be to either 1) prove that an 
idea of concept works, much like an experiment, or 2) show how 
something works or performs, much like a presentation. Whereas the 
first purpose implies the development and testing of new ideas, the 
latter purpose stresses replication of existing ideas in new contexts. 
Program regulations for the Cooperative Demonstration Program do not 
specify which of the two (or both) purposes is to be addressed by the 
grantees and applicants are free to select which emphasis they wish to 
pursue. This part describes two aspects of projects which may reflect 
the testing of new ideas: the high technology aspect of each project 
and the innovation, if any, in vocational education services. 



L Definition of High Technology 

The "high technology" characteristics of the nine Cooperative 
Demonstration Grant projects varied. They included: 



» Training d esigned to prepare students for 
■lobs in fields gene rally held to be "high 
tech" In that thev manufacture hioh 
technology products . The training itself 
might be sophisticated or basic and lead to 
entry-level jobs or to upgrading the skills 
of current workers. For example, an 
electronics course for high school students 
led to an entry-level placement in a laser 
optics manufacturing plant; 

■ Training to enable students to use high-tech 
eouloment even thou gh the field for which 
they are training i s not generally considered 
to be high tech . Examples Included training 
in the use of computer assisted drafting, 
automotive repair (which includes high-tech 
diagnostic and other equipment), film 
industry technicians; 

■ Training using computers regardless of the 
field for which training is received . For 
example, students In one program were trained 
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to use word processing packages and Lotus 1-* 
2-3; and 

■ TnlT^lTiq ip $M1H n pr^p^rnipn f^r 

specific occupational training for a high 
technology field . In one project, 
postsecondary students who wished to enter 
training in a high tech field but who lacked 
«dequate math skills were provided with 
remedial training; in another, secondary 
students were offered an introductory course 
in materials science technology. 

All projects were not equally "high-tech." Not all training using 
computers was high-tech, unless the training also included learning 
about computers or using the computer to learn a high technology 
application. Remedial or other basic skills preparation not geared 
directly to preparation for high-tech jobs appear to be tangential (at 
best) to the intent of the Act. If this program Is continued, Federal 
officials may wish to define high-tech fields they wish to involve in 
the training or the high technology activities they would permit. 

L Extent of Innovation 

Although the FY1988 program regulations and application materials 
do not require projects to be innovative (nor is the term "Innovative" 
even used), the study*s Advisory Panel and staff felt that It was 
important for the demonstrations to advance the knowledge of the 
vocational education field by testing new Ideas. At each project 
visited, the study team sought to understand what, if anything, the 
project saw as innovative or new In its offering. Much of what was 
observed during the site visit was hardly "cutting edge" with respect 
to instruction, partnership, supplementary or coordinated services, or 
serving special populations (as defined by the Perkins Act or 
otherwise) . 

The absence of innovation within and across projects might have 
been due to the absence of a priority for it in the regulations or 
points awarded for it in the scoring of applications. Lack of 
innovation might also be due to the short time frame of the grant— it 



67 

is hard to start and complete an innovative project in 18 months. 
Still, a few projects did manage to experiment with new designs, 
services, or partnerships. 

One innovator was Moorpark College, who':, project developed a 
model of cooperation among a community college and a group of high 
schools to promote technology achievement and career among low- 
achieving ("at-risk") students. Important elements of the project 
included: 



■ Constructing a curriculum and cooperating 
with high school teachers so that instruction 
in high schools and the community college 
were synchronized; 

■ Training teachers 1 the suironer so they would 
feel comfortable w* , the curriculum, e.g., a 
math teacher learned about eleccronics; 

■ Enlisting the high school teachers in the 
consnunity college instructional process and 
vice versa; and 

■ Using vocational interests as a means to 
teach academic skills (and informal 
vocational setting and approach as well). 



At the same time, the project encountered difficulties that are often 
faced when institutions with different goals try to cooperate. The 
lessons learned include that: 



i The college wanted "at-risk" students but 
some school districts instead sent their 
brightest— the result was a class composition 
at the community college that required a 
large amount of individual instruction and 
low achieving students was not always well 
served; 

■ Costs of moving students from high school to 
community college and back were great; 

■ There was no clear role for the private 
sector in this project as the opportunities 
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for jobs in laser optics failed to 
materialize and most students chose to attend 
community college after graduation. On the 
other hand, the project director reported 
that these students are remaining at the 
community college longer than students 
typically do and she attributes this to the 
"head start" they got in meeting the staff 
and feeling part of the institution; and 

■ The college and school districts involved 
purposely did not use vocational education 
teachers at two of the high schools because 
of "turf issues. The other two schools did 
not have electronics programs so there was no 
issue. This does not bode well for efforts 
at vocational /academic integration that 
involve vocational high school instructors. 



In lighu of current interest In using vocational education as a means 
to engender educational interest and teach basic skills, the Moorpark 
project might be worthy of further dissemination. Furthermore, the 
"hands-on" physics course created by Battel! e and Richland School 
District is also worthy of attention, even though it was not developed 
under this grant. 

Although some projects were not innovative, they benefitted those 
students involved. Most of the projects addressed needs in their 
communities— opening up job opportunities in growing fields, upgrading 
the skills of workers, exposing small manufacturing plants to the 
opportunities of using high technology equipment, and increasing the 
capacities of training institutions and their staff to deliver training 
in high technology fields. Project staff cited three kinds of 
beneficiaries: trainees, employers, and the institutions providing the 
training. 

Finally, what do these projects teach us about public/private 
partnership as a means to foster training and innovation? Partner 
roles were limited. Several partnerships failed to live up to 
expectations. In at least three projects (Skyline, Indiana University, 
and Moorpark) the private partners appear to have made inflated 
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promises about what they could deliver/ When forced to deliver jobs 
or support services, they were unable to do so, putting these projects 
at risk. 



In the case of Indiana University, the private sector partner discussed 
here is the economic development agency— an organization developed by the state— 
which was supposed to link the institution to small manufacturers in the service 
area. 
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F. Issues to be Addressed In the S econd Year 



These findings about project implementation raise issues that 
should be explored in the survey and case studies in Year 2 of the 
study. These issues are in addition to issues of partner role, 
innovation, and grant time frame, etc. already discussed. 

Continuation after the Grant : The terms of the grant award 
required grantees to draft dissemination plans. Such plans imply that 
projects will live beyond the end of the Federal grants. Although some 
projects may not intend to continue, those that do (or that expect to) 
disseminate methods or products should be taking appropriate action. 
Five of the nine projects requested grant extensions (for a maximum of 
three months), usually to allow them to complete an evaluation or 
curriculum development. 

The study team should look at whether poorly-designed or executed 
plans were revised after the end of the Federally funded project. For 
example, the Skyline project finally resolved scheduling problems with 
Toyota Motor Sales just as the Federal grant was ending. The project 
director noted that only at that point did he feel he had the freedom 
to deviate from the original Toyota approach. 

Appropriateness of Customized Training : The study team should 
clarify if the intent of tie Cooperative Demonstration Program is to 
train employees of individual i-^ployers. The Perkins Act specifically 
precludes awards to private companies. Even assuming that 
demonstrating employer-specific training is a Federal goal, what types 
of employers, or employees, should be priorities under the Cooperative 
Demonstration Program? Should employers be producers of high-tech 
products? Should the employees be those most in need of skill 
upgrading (or those most disadvantaged?) within those firms? 
"Customized training" is a de facto priority of the Program without any 
acknowledgement in regulations or grant rules. 

Evaluati nr^: The FY1988 projects visited by the study team were 
just beginnir their evaluations at the time of the site visits. Few 
had planned for evaluation and infused evaluation activities into the 
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course of the grant. FY1989 grantees, on the other hand, received 
instructions in conducting evaluations at their first organizational 
meeting (when their grants began). The study team will explore whether 
the greater Federal emphasis on evaluation in FY1589 is reflected at 
the project level . 
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IV. PROJECT COSTS AND BENEFITS 



The third question answe'.'-d by the evaluation is whtethe«^ project 
costs are "reasonable" in relation to the projectrrd or actual outcomes 
of the project. Project costs s.re defined as the Cooperative 
Demonstration Program grant plus the non-Federal cash aud/or in -kind 
matching contributions provided by the grantees. The outcomes of the 
project are defined as the numbers of students trained, the number of 
staff trained (if the project also focused on ataff deve1op»3nt), and 
the number of course hours developed {If the project also focused on 
curriculum development). The judgement of "reasonableness* is made by 
comparing costs and benefits across the projects rather than comparing 
them to some absolute standard. 

While not yielding an absolute cost benefit .atio for each major 
outcome (as in traditional cost benefit analyses), these analyses 
quantify the major outcomes of the projects and, where possible, 
calculate the total costs incurred in achieving each of those outcomes. 
No attempt is made to assign a monetary value to the benefits resulting 
from the project outcomes (e.g., the dollar value of learning a new 
skill), but every effort Is made to make appropriate comparisons 
between project treatment costs and outcomes. 

This :«ction Is divided Into four parts. Part A identifies the 
major cost and benefit Issues to be addressed and the operational 
definition of those issues. Part B defines and enumerates the project 
costs, and Part C defines and enumerates the major project benefits. 
Part D then compares costs with benefits in accordance with the major 
issues raised In Part A. Finally. Part E briefly describes changes in 
the plan for the cost benefit analysis In Year 2. 
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A. Major Issues 

Project costs and outcomes first are aggregated to overall cost 
estimates and benefit estimates. These are used to compute four 
measures: 

■ Treatment costs : total project costs minus 
planning and development costs; 

« Project intensit y: total treatment hours 
divided by the number of successful 
completions; 

1 Average unit cost of de livered services: 
total treatment cost divided by total units 
of service; and 

■ Service cost per unit of outcome: total 
treatment costs divided by the number of 
successful completions. 

The definitions of treatment hours, successful completions, and units 
of service produced will vary according to the type of treatment (e.g., 
training versus curriculum development) implemented by the project. 
For projects that focus on student training, the number of treatment 
hours is the total number of classroom contact hours, the successful 
completions are those students finishing the class, and the units of 
service are the numbers of hours of training per student. 

The analysis of the four major issues is based only on the FY1988 
projects for which data were collected during the site visits. The 
eight FY1988 projects visited were selected "according to their ranking 
in a preliminary assessment of evaluability. Because of the lack of 
data regarding project intensity (defined as treatment hours divided by 
successful completions}, the study team could not select the sites for 
variations in treatment intensity as originally proposed. However, the 
team could collect limited cost and outcome data (e.g., project budgets 
and numbers of students trained) from all eight projects. These data 
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were collected through face- to- face interviews and reviews of budget 
and expenditure reports. 

During three-day site visits, team members collected data from the 
project's records, the grantee's records, partner's records, and 
project staff interviews. Project records yielded all direct 
expenditures from the grant and from non-grant sources. The grantee's 
records documented all overhead expenditures or other funding sources 
within the organization. Partner organization members were interviewed 
to determine direct exenditures or in-kind contributions to the 
project. Finally, project staff were interviewed to identify other in- 
kind contributions to the project, such as donated equipment or release 
time from other organizations. Although in-kind contributions cannot 
be precisely translated into dollars, the analysis used general 
estimates of the value of the contribution stated in the grant 
application. 
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B. Pro.iect Costs 

Project costs are the sum of all resources— either financial or 
non-financial— used for project activities. If a project uses only 
funds provided by the Federal grant, then project costs are equal to 
the grant amount. Cooperative Demonstration grants, however, require 
grantees to contribute at least 25 percent of the total project costs. 
Thus, project costs consist of both the Cooperative Demonstration grant 
amount and grantee match. 

Project costs often include more that what is paid for from 
Federal funding and matching local resources. Projects may use 
existing instructional services paid by the grantee as part of its 
regular operations. Or projects may use the staff and materials funded 
by another, parallel grant to supplement the activities of the 
Cooperative Demonstration project. These outside sources of support 
for the project should be included in the calculations of total project 
costs in order to present an accurate picture of the project's true 
resources. Thus, 

total project costs « Federal grant + local match + outsid** 

project resources. 

Unfortunately, outside sources of support often cannot be included in 
the calculation of total project costs because they are not explicitly 
tracked by the grantee. Although the team was not able to account for 
all the costs of thp project*s activities, it is still possible to 
perform a limited assessment of the direct services and materials paid 
for with grant funds. 

1. Cost Categories 

The primary components of project costs used ir the analysis are 
those found in the line item budgets of the projects. The line items 
include: 
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■ Salaries and Wages; 

■ Fringe Benefits; 

■ Travel ; 

■ Equipment; 
8 Supplies; 

■ Contractual Services; 

■ Other Costs; 

t Total Direct Costs; and 

■ Indirect Costs. 

These line items are explained in more detail fn the following 
paragraphs. 

Salaries a nd Wages . Primary staffing costs are salary and wages 
paid to staff, including annual salary and hourly wages for all 
employees of the grantee— or the partner organization— who work on the 
project. Among the staff included in the salaries and wages category 
are: teachers, instructors, administrators, other certified personnel, 
clerical staff, and support staff. 

Fringe Benefits . Employees usually received fringe benefits as 
part of their compensation package Including sick leave, annual leave, 
holidays, health insurance, etc. Fringe benefits are usually 
established by the institution as a percentage of total salaries and 
wages. 

Travel . Travel costs, if any, are listed as a separate line item, 
including air or train fare, car rental, ground transportation, hotels, 
meals, and tips. 

Eouioment . All equipment purchased by a project is a separate 
category. Although the Program discouraged FY1988 grantees froni using 
grants to purchase equipment and prohibited it in FY1989, some projects 
did purchase new equipment. Other projects ma: have used other non- 
Federal sources of funds to pay for project equipment. Among items 
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that might be purchased are specialized electronic equipment, 
manufacturing machinery, or computer hardware. 

Supplies . Projects may require routine office supplies or may use 
specialized materials; these are a separate budget category. Among the 
types of supplies a project may use are office supplies and 
instructional materials. 

Contractual Services . Projects may hire outside experts or 
additional temporary personnel. For example, projects way hire outside 
evaluators to evaluate the project and may use specialized services 
provided by other companies. These experts and services are paid 
through contracts that specify the work to be accomplished and the 
charges for that work. The rules governing the use of consultants 
(individuals) are different from those governing the use of companies 
(subcontracts), but both Involve the external acquisition of services 
and are grouped in one category. 

Other Direct Costs . Direct costs that do not fit Into these 
categories are listed as "other costs" amd may Include space rental, 
telephone, and postage if they are not paid for Indirectly (see 
"indirect costs" below). 

Total Direct Costs . The sum of all the direct cost categories 
yields the total direct costs of a project. 

Indirect Costs . Indirect costs are charges to the project made by 
the grantee institution for overhead Items such as office space, heat, 
electricity, postage, accounting services, and iranagemer.t services. 
The grantee Institution usually provides these services and supplies to 
all projects and programs. To pay for ite; that are difficult lo 
itemize, the grantee institution charges each project an Indirect cost 
or overhead rate. The indirect cost Is usually based on a percentage 
of the total salaries and f*"1nge b'insflts, but iray also include other 
direct costs as well. Under current grant regulations, grantees arc 
allowed to charge a maximum indirect rate of eight percent. 

Total Prcjgct Costs . Total project costs are the sum of all 
direct cost.i and all indirect costs associated with the project. 
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2, Sources of Funding 

Projects fund activities from three major sources. First, 
projects use the grant funds provided by the Federal government through 
the Cooperative Demonstration Program grant award. Second, projects 
use local resources from public or private organizations (e.g., the 
grantee or partner organization) in a required 25 percent "local 
match." Program regulations require grantees to contribute a minimum 
of 25 percent of the total project costs. The contribution of the 
grantee can be composed of cash or In-kind resources from the grantee, 
the partner organization, another state or local program, or another 
Federal program. In-kind resources include grantee staff time, partner 
staff time, equipment, facilities, services, and materials. 

Third, projects use funds or in-kind contributions not identified 
in the grant application nor reported in the project expenditure 
repor-s. These mostly are underlying instructional or support services 
provided by t!ie grantee as part of the regular educational program. A 
second source are other projects operated by the grantee which are 
relevant to the Cooperative Demonstration project and which provide 
services, equipment, or other support contributing to the success of 
the funded project. The grantee may not Identify these resources with 
tha project because 1) the grantee already has satisfied the 25 percent 
match, 2) it would require additional grantee resources to track these 
contributions, 3) the grantee may need to use the "local match" in 
excess of the required 25 percent contribution to offset any matching 
fund-J disallowed in a subsequ?>nt Federal audit of the project, or 4) 
the addit"!onal resources are too difficult to handle in the accounting 
system. Although these other resources are not tracked in the grant, 
they are still part of the overall costs of the project and are 
relevant in determining the reasonahleness of project costs. 

The total costs of each project and sources of funding are 
presented in Table 16. Total project costs ranged from a low of 
$308,335 for Northampton Commumty College to a high of $753,842 for 



Table 16 
TOTAL PROJECT COSTS 



i PROJECT 


nOtML fiSANT 

f\ms 


iOCAL CASN 
MTCH 


LOCAl 1N-KIM> 
NATCN 


Qim nUMSCT 


TOTAL MtOACf 
OJSTS 




$388,770 




$ 48,749 


I 0 


$530,519 


Moorpark 


299,564 


69.696 


77,889 


0 


447,149 1 


Skyline 


191,775 


50,986 


127,304 


5S.OO0 


425,065 


Valencia {Bfg.) 


222,456 


130,034 


71,860 


0 


424,550 


Valencia IfHml 


406,517 


52.065 


117,617 


0 


575,999 


Northampton 


227,584 


63,169 


17,582 


0 


308.335 


Indians U. 


213.040 


68,523 


154,00) 


0 


435.563 


Kichland 


322,267 


10,575 


427.000 


0 


759.842 



* total project costs » Federal grant + local cash match + local in-kind match 

+ other project resources. 
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Richland Public Schools. Only one project, Skyline Community College, 
received other grant? which directly contributed to the Cooperative 
Demonstration project but which were not reported as part of the local 
match. The percent of total project costs contributed by the grantee 
ranged from a low of 26.1 percent by Northampton Coamunity College to a 
high of 57.6 by Richland Public Schools. Thus, the district with the 
lowest total project costs also contributed the least while the 
district with the highest total project costs contributed the most 
local match. 
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Project benefits are outcomes that improve the ability and/or 
eaployment status of students or the quality of the vocational 
education process. These outcomes can be expressed in either 
quantifiable terms (e.g., the number of students successfully 
completing training) or non-quantifiable terms (e.g., a new technique 
for skills identification). Although interesting, non-quantifiable 
outcomes are difficult to measure and to express In terms that would 
allow comparison with project costs. Consequently, the cost benefit 
analysis measures and assesses only quantifiable outcomes from the 
projects. 

Quantifiable outcomes vary with the type of activity and the focus 
of the project. For example, the outcomes and benefits from curriculum 
development are likely to be new teaching modules or materials. 
Outcomes and benefits from a dissemination project are likely to be 
"how- to-do-it" descriptions of the project, formal publications and 
presentations, or greater awareness by other vocational education 
institutions. Similarly, the outcomes and benefits of training, with 
some student assessment, are the number of students with documented 
improvement in skill levels. 

The Statement of Work (SOW) for this contract identified three 
quantifiable outcomes to be measured for each project: 1) the total 
number of service hours, 2) the number of successful participants, and 
3) the number of person hours of services received. Because tne 
primary purpose of the projects is to improve vocational education 
through improved student training, most services are student training. 
Therefore, primary outcome measures were: 1) the total number of hours 
of student training provided, 2) ihe number of students completing the 
training, and 3) the number of hours of training received by each 
student. 

Other activities are also conducted under the grant, however, and 
these provide services ?» well. Their outcomes can be measured by 
refining the three general measures. For example, the outcomes of 
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orojects emphasizing staff development can be expressed as: 1) the 
total number of hours of staff training provided, 2) the number of 
staff completing the training, 3) and the number of hours of training 
received by each staff roember. Similar outco<nes can be constructed for 
student assessment and the other activities. The following are the 
quantifiable outcomes, by type of activity, that are measured for each 
project in the cost benefit analysis: 

Activity MfifiSSS 

student assessment number of students assessed 

training students number of students completing 

training 

staff development number of staff trained 

curriculum development number of course hours developed 

slcills identification number of industry skills 

identified 

dissemination/diffusion number of other end users 

adopting product 

partnerships/networking dollar value of partner(s) 

contributions 

The major outcomes from each type of activity were described in Section 
III and are summarized in Table 17. 

Outcome measures do not measure outcomes. Many impo**tant 
outcomes and benefits may not be measureable until after the 18 month 
grant period. For example, while the number of students hired is one 
measure of the quality of training, it may not be possible to collect 
this data If the students are years away from jobs or they were already 
employed. The demonstrated mastery of technical skills will be the 
Ultimate measure of the effectiveness (along with job placement) of a 
new curriculum, but the project may not conduct pretests because no 
tests exist yet. Thus, the outcome measures may tell what was done, 
but not how well It was done. 
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Table 17 
SWWARY OF PROJECT BENEFITS 





STUDfMT 


TMimifG 


STAFF 


DEWIOPNENT 


10€iinFSC. 


OlSSEMid- 
ATIOH 


FAflTf^RSHIP 1 




o instructof s 
adBin. pre 
and post 


o 450 students 

trained 
o 1,032 hoirs 


o SO tt^Hers 

trained 
o 20W hours 


0 41 cotn^se 
aodules 


o surveyed 
285 firm 
o ^ interv. 


o newletter 
o conference 
presentat* 


0 16 second. | 
& post-sec. 1 
agreements | 




o pre and 

post tests 
0 skill tests 


o SS students 

trainer 
o 23^936 hMirs 


o 3 tethers 

trained 
o 360 hours 


o ffiodif led 
exist ii)^ 
curricutiM 


o esti8»tes 
froe 2 
f ims 


0 conference 
presentat . 


o BLC 1 
o Ventura | 
hi^ schls* 1 


Skyline 


0 €n9, and 

fneifi tests 
o on-the*Jc^ 


o 17 stuttents 

trained 
o 40«500 hours 


0 7 tethers 

trained 
o ^ hours 


0 deyel(^9ed 
new tests 
& 4al»heets 


c worked with 
local 
<tealers 


0 conference 
presentat * 


o Toyota i 
t'fen pro9. | 
oil dealers | 


Valencia tmfg.} 


0 instructors 
ddk»in« pre 
and post 


o 392 students 

trained 
o 2S,480 hours 


0 none 


o developed 
sotoe nen 
ntsteri als 


o none ' 


o confer^ice 
fH^esmtat « 
o St* cofna* 


o Strofflberfi- 

Cerlson 
o contribute 


Valencia tfil»} 


0 pre and 
post tesis 


o 131^ stucfents 

trained 
o 13^5{K) hours 


0 none 


o new curric* 
ui\m in S 
areas 


o none 


o videot^ 
o materials 
to IRiG 


o Universal 

Studios 
o vendors 




0 none 


o 233 stud&its 

trained 
o 20,136 hours 


0 seminar for 
SNT teachers 


o 7 new 

courses 
o 4 vp^r&ded 


o done by 
conf>^ 


o national 
teleconf . 
o marketing 


0 renewed 
relatlors 
w/ clierits 


Indiana U. 


o ai discre- 
tion of 
tnstriictor 


o 648 slud^ts 

trained 
c 12, 9« hours 


0 nc-» 


o some modil* 
for CCW 
students 


surveyed 
^tm firm 
o follow-!^ 


Q onference 
presentat* 


0 11 firms 
o local voc. 
schools 


1 Xichl^^ 


o baseline 
md end 
surveys 


o 237 students 

trained 
o 42^660 hours 


o 15 teachers 

trained 
o 2,^QQ hours 


o formalized 
previous 
curricutuo 


o done by 
local adv. 
com! t tee 


o conference 
^esentat , 
o 2 articles 


o Battel le 
o 6 schools 1 
l^lement | 
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D. Cost/Be nefit Analyses 



Ideally* the study would match costs to specific activities and 
outcomes so that it could determine cost effectiveness. However, it 
was not possible to collect cost data for all outcomes for two reasons, 
first, projects do not code their expenditures by activity and are not 
equipped to account for costs at that level of detail. Their internal 
accounting systems tend to reflect aggregate expenditures according to 
budget line items. Second, several activities may contribute to the 
same outcome or objective. The projects Implemented a set of 
activities and often one activity Is used to support more than one 
objective. 

The analysis focused on only those activities for which cost data 
were available through the grantees accounting system or the final 
contract budget: planning and administration, student training, and 
curriculum development. Planning and administration costs include the 
wages of the project director -and project clerical staff, associated 
fringe benefits, other direct costs associated with r.dmini strati on, and 
indirect costs. Student training costs include the wages of 
instructors and other specialists, associated fringe benefits, other 
direct costs (e.g., textbooks, supplies, travel, and stipends), and 
indirect costs. Curriculum development costs include the wages of 
instructors and curriculum development specialists, associated fringe 
benefits, other direct costs (e.g., training workshops, travel, and 
printing), and indirect costs. 

1. Treatment Costs 

The first major analysis separates planning costs from the costs 
of providing the service. The planning and administrative costs are 
defined as the sum of the wages of all administrative, clerical, and 
other non-instructional project personnel, plus any non-instructional 
direct costs. The planning and administrative costs subtracted from 
the total project costs to yield the total treatment costs: 
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treatment costs » {total project costs) - (project director wages 

+ clerical wages + fringe benefits + other 
direct costs + related indirect costs 

The treatment costs and planning/administration costs for each project 
are displayed in Table 18. The proportion of total project resources 
devoted to services instead of administration ranged froa a low of 64.8 
percent in Moorpark to a high of 97.6 percent In Richland. The 
administration costs for Richland appear artificially low because they 
do not Include the cost of aJroini strati on by partner schools. 

2. Project Intensity 

The second analysis uetermines the Intensity of the project (i.e., 
the number of successful outcomes for each service relative to the 
effort expended to accomplish those outcomes) as: 

project intensity » total treatment hours 

number of successful completions 

The effort to train students Is the sum of the hours students spent in 
training across all courses. The number of successful outcomes Is 
defined as the sum of the students completing the training. In 
projects where the treatment was staff training, the number of 
successful outcomes is the number of teachers completing the training. 
The project Intensity for the two major activity groups for which data 
were available— training students and training staff— are displayed for 
each project in Table 19. The amount of training per student completer 
ranged from a low of 2.3 hours at the University of Wisconsin-Stout to 
a high of 1,280 hours at Skyline Community College. Staff training 
ranged from 30 hours per teacher to 160 hours per teacher. 

3. Average U nit Cost of Services 

The third analysis determines the average cost per unit of 
service. The total cost of the service is the sum of all costs 



Table 18 

PLANNING/ADMINISTRATION COSTS VERSUS TREATMENT COSTS 



i PBOJECT 


TOTAL I^MUECT 


PLAiWIim MO 

Mmi». mis • 


PERCENT 


TREATJCHT COSTS 


i>ERC£NT i 


1 U. Wisconsin 


$530,519 


$101,762 


19.2 


$4M.757 


80.8 


MtKM-park 




157,185 


35.2 


289.%4 


64.8 i 


Skyt ine 


467.453 


139,927 


30.0 


327.526 


70.0 


Valencia Imig.} 


424,350 


100.185 


23.6 


324,165 


76.4 


Valencia ffilm) 


575.999 


70,276 


12.2 


505,723 


87.8 


Northaaptwt 


3tM,335 


59.183 


19-2 


249,152 


80.6 


Indicia U. 


435,565 


104.691 


24.0 


330,872 


76.0 


Richland 


759.842 


18.353 


2.4 


741,489 


97.6 



* planning and administration costs « project director wages + clerical wages ^ fringe benefits 
other 

direct costs + related indirect costs 



Table 19 
PROJECT INTENSITY 



i PMMECT 


TOTAL mats 

STU^NT 
TRAINING 


MMSER OF 

STUDENTS 

COmETINa 


PROJECT 
INTENSITY* 


TOTAL mm$ 

STAFf 
TRAINING 


mmm of 

STAFF 

COW>i£TIMG 


raojECT 1 

INTQiSITY 


i U. Uiscofisfn 


1,0J2 


450 


2.3 









Motwpark 


23.936 


65 


181.6 


120 


4 


30.0 


1 Skyline 


21,760 


17 


1.280.0 


200 


2 


100.0 


1 Valencia tafg.) 


25,480 


392 


65.0 








Vatencis lUiftil 


102,060 


135 


756.0 








Uorthmpton 


174,72* 


233 


749.9 








Indiana U. 


12,960 


418 


31-0 








Richland 


42,660 


237 


180.0 


24(» 


15 


160.0 



* project Intensity « total activity hours 

number of completers 
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associated with the direct provision of the service, excluding project 
planning costs: 

average unit cost « total treatment cost 

of service total units of service provided. 

For example, total service costs for providing student training would 
be the sum of the project costs for staff, materials, overhead, etc., 
for all the courses. The total service costs for curriculum 
development would be the sum of project costs for staff, testing, 
reproduction, overhead, etc. for all the courses produced. The total 
units of service provided is the sum of all students entering training 
or new courses. 

The average unit costs for student training and curriculum 
development for each project are displayed in Tables 20. The average 
unit cost for student training ranged from a low of $0 at the 
University of Wisconsin-Stout (all training costs were born by the 
participating schools as part of their regular teaching 
responsibilities) to a high of $25.53 at Indiana University. 

It should be noted that comparing the average cost per hour of 
training across projects may create an unfair comparison because of the 
variations in the intensity of the training and the number of students 
being trained. The average cost for an hour of training at Indiana 
University was the highest of the projects, but that project also 
trained the second highest number of students. To provide a more 
accurate comparison, the analysis should compute the average cost per 
hour of training oer student trained . This can be operationally 
defined as: 

average unit cost » total service costs 

of service per total units of service provided 

unit of outcome number of completers 

Table 21 shows the per hour per student costs for the projects. 
The per hour per student costs range from a low of $0.00 for the 



Table 20 
AVERAGE UNIT COSTS OF TREATMENT 



1 PftOKCT 


TOTAL m»i 
OF STUOeiiT 
TMINIMC 


TOTAL OSTS 
FOt STUDENT 
TMININS 


C£»T PER 

mst Of 

TftAINIIIG * 


TOTAL imffiS 
Of CMRICULA 
I^VELOPEP 


TOTAL CCST$ 
FOR CLOUIIC. 
DEWlOWtfUT 


COST PIER 1 

must 1 


U. Wisconsin 


1,052 


% 0 


0.00 


492 


»428,5S8 


S871.0S 1 




25,936 


289,964 


12.10 








1 Sky tine 


21,760 


S27,531 


«.11 








i Vfiteocia Imfg.] 




469,723 


18.43 








1 Valencia {film) 


102,060 


317,577 


3.11 








Northaopton 


174,724 


107.271 


0.61 








Indians U. 


12.960 


330,872 


25.55 








tichtand 


42,660 


374, «)0 


8.76 


1.440 


361,045 


250.73 



o 



* average unit cost «= total treatment costs 

total units produced 

1 
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Table 21 

AVERAGE UNIT COSTS PER UNIT OF OUTCOME 



i f>fiOJECT 


TOTAl MXiKS 
OF STUDENT 
TRAINING 


TOTAL Q»TS 
F08 STUDENT 
TRAINII^ 


COST PES 

muxi OF 

TRAINING 


NU»a£R OF 
COI«>LETEKS 


COST 

mm Kit 

STl»ENT * 


1 U. Wisconsin 


1,0J2 


S Q 




450 




1 Moorpark 


ir3,956 


289,^ 


12.10 


85 


0.142 1 


Skyline 


21,760 


527,531 


15.11 


17 


0.889 1 


Valencia fmfs.l 


25,*ao 


469,725 


18.43 


392 


0.D47 


Valencia tfitsi] 


102,060 


317.577 


3.11 


155 


0.023 


Noi-thaopton 


174,724 


107,271 


0.61 


253 


0.003 




t2,%0 


530,872 


25-53 


418 


0.061 


fiichland 


42,660 


374,0M> 


8.76 


237 


0.037 



* average unit cost « total treatment cost 

of service per total units of treatment provided 

unit of outcome number of completers 

P8 
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University of Wisconsin-Stout to a high of $.89 at Skyline Community 
College. The costs for the remainder of the projects tend to 
concentrate between $.01 and $.06 per hour. Skyline's costs are out of 
proportion with the other projects because so few students were hired 
into the program by the Toyota dealers and the program operated at 
level far lower than originally planned. 

4. Service Cost oer Un it of Outcome 

The fourth analysis determines the service cost per unit of 
outcome. As In calculating average unit costs, the total service cost 
here is the sum of all costs associated with providing the service. 
Thus, service costs can be expressed as: 

service cost " total service costs 

per unit of outcome number of successful completions. 

The number of successful completions is defined as the sum of the 
students cr completing the training and courses successfully developed. 
The unit service costs for training students and curriculum development 
are displayed for each project in Table 22. Service costs ranged from 
a low of $0 at University of Wisconsin-Stout to $19,266 at Skyline 
Community College. Again, Skyline's costs are out of proportion with 
the other projects because the project trained fewer students than 
originally planned. 

In summary, the answer to the question "are project costs 
reasonable in relation to project outcomes?"* appears to be yes for all 
project except Skyline. The per unit and per outcome costs for all the 
other projects tended to cluster in the same area even though total 
costs and project intensity varied substantially. Skyline's costs, 
although understandably high, would suggest that the project was 
relatively unsuccessful as a model for other vocational education 
Institutions. 
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Table 22 

SERVICE COSTS PER UNIT OF TREATMENT 



1 FSOJECT 


TOf Ai COSTS 
rat STUMENT 
TIUIitING 


COMPLETERS 


SERVICE 
CtST PER 
STUWEMT * 


TOTAL COSTS 
ftX CtfltfilCUlA 
PEVEIOPED 


MJNSiR OF 

COURSES 

DEVELOPED 


OJST PER 1 
COURSE 1 


1 U. Wiscorttin 


S 0 


450 


S 0 


S428,SSa 




$ 10.453 i 


1 ftoorpark 


289.964 


85 


3.411 








1 Skyl ine 


327,531 


17 


19,266 








1 Valencia (oifs.I 


469,723 


392 


i,m 






1 


Valefvia IfiloO 


$17,577 


155 


2,352 








Northanpton 


107.271 


233 


460 








Indiana u. 


330,672 


418 


792 








Richiand 


574,000 


257 


1.578 


361,045 


8 


45,130 



* service costs «= total tyeatroent costs 

number of completers 
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E. Plan? fpr Ygar Z 

The nine projects to be used In the cost benefit evaluation in 
Year 2 of this study will be selected from anjong the 15 FY1989 projects 
ranked highest in the evaluability assessntent. The 15 projects will be 
further ranked according to the intensity of treatment (i.e,, total 
treatment hours per completer). Three projects representing high 
intensity treatments, three projects representing medium intensity 
treatments, and three projects representing low intensity treatments 
will be selected. The nine sites will be visited to collect the 
necessary cost and outcome data. Cost and outcome data also will be 
collected from the other six projects visited. As with the FY1S88 
projects, conclusions about the FY1989 project will be generalized only 
to those projects for whom data was collected and not to the universe 
of 30 FY1989 projects. 

In Yea** 2 of the study, cost and outcome data will be collected 
using field protocols similar to those used in Year 1. Cost and 
outcome data also will be collected through a review of project 
progress reports and during the mail survey of all Year 2 projects. 
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V. CONCLUSIONS ANO RECOMMENDATION S FROM THE FIRST YEAR 

A. Summary of Findings 

Section I of this report identified three evaluation questions to 
be answered by the study. First, do the grant applications present a 
clear and coherent design for a project? Second, were the grantees 
able to ijuplement their project designs as proposed, and, if not, what 
problems prevented that implementation? Third, are project costs 
reasonable in relation to projected or actual outcomes? Each question 
was the subject of a separate section of this report. This part 
summarizes these findings and presents the answers to the three 
questions. 

1. Do the grant applications submitted for funding present a c lear and 
coherent de sign for a project? 

The terms "clear" and "coherent" were operationally defined to 

mean "logical" and "plausible" respectively. When taking both factors 

into account, three projects were predicted to be successful 

demonstrations; ten projects were predicted to be potentially 

successful; and ten projects were unlikely to be successful. Thus, 13 

out of the 23 applications presented clear and coherent design for 

their project. Among the eight projects selected for site visits, the 

final rankings were: 







Project 






Project 


Plausi- 


Predicted 


Grantee 






$MCC9$5 


Valencia Community College [mfg.] 


high 


high 


successful 


Moorpark College 


high 


high 


successful 


Richland Public Schools 


high 


medium 


successful 


Skyline Community College 


medium 


mediiuB 


somewhat 


Valencia Community College [film] 


medium 


medium 


somewhat 


Indiana University 


medium 


low 


less likely 


Northampton Community College 


low 


high 


less likely 


University of Wisconsin - Stout 


low 


low 


less likely 




t oo 
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The predicted success of the project based on the application's 
clarity and coherence can be compared with the actual success of the 
project based on the numbers of students trained. Among the three 
projects most likely to succeed, two projects (Valencia [mfg.] and 
Richland) fell short of their targeted number of students to be 
trained. Among the two projects (Skyline and Valencia [film]) somewhat 
likely to succeed, both projects met their training target, although 
Skyline had a much lower participation rate that initially expected. 
Among the three projects less likely to succeed, two projects 
(University of Wisconsin-Stout and Northampton) fell short of their 
targeted number of students to be trained. It appears that the clarity 
and coherence of the application Is not a good predictor of the likely 
success of the training component of the project. Other aspects of the 
project, e.g., long term Impacts on the labor market or numbers of 
students getting jobs because of the training could not be evaluated 
within the time-frame of the current study. Therefore, the usefulness 
of application clarity and coherence as predictors of long term success 
is still unknown. 

2. Were orantees ablg to Implemen t their project designs as proposed? 

The success in implementation was measured by the extent to which 
the project met its goals and objectives regarding student training, 
partnership development, and curriculum development. Among the eight 
projects visited, the outcomes were accomplished as planned at those 
projects designated with a "yes": 



Grantee 



Student 
Training 



Partner 
ship 

Pevel f 



Curriculum 

Develop- 

ment 



Valencia Community College [mfg.] 
Hoorpark College 
Richland Public Schools 
Skyline Community College 
Valencia Community College [film] 
Indiana University 
Northampton Community College 
University of Wisconsin - Stout 



yes 
yes 
yes 

no 
yes 

no 
yes 
yes 



yes 
no 

yes 
no 

yes 
no 

yes 

yes 



yes 
n/a 
n/a 
n/a 
n/a 
yes 



n/a 
n/a 
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Thus, five of the eight projects were able to implement their proposed 
plan. 

3. Are proiect costs reasonable in relation t o project outcomes ? 

The answer appears to be yes for seven of the eight projects (the 
exception is Skyline). The per unit and per outcome costs for all the 
other projects tended to cluster in the same area even though total 
costs and project intensity varied substantially. Skyline's costs, 
although understandably high, would suggest that the project was 
relatively unsuccessful as a model for other vocational education 
institutions. 

The proportion of total project resources devoted to project 
planning and administration ranged from a low of 2.4 percent in 
Richland Public Schools to a high of 35.2 percent. The costs for 
Richland appear artificially low because it does not include the cost 
of administration by partner schools. 

The second cost issue was the intensity of the project (i.e., the 
number of successful outcomes for each type of treatment relative to 
the effort expended to accomplish those outcomes). The amount of 
training per student completer ranged from 2.3 hours at the U. of 
Wisconsin to 1,280 hours at Skyline Community College. Staff training 
ranged from 30 hours to 160 hours per teacher. 

The third cost issue was the average cost per unit of treatment. 
The total cost of the treatment is the sum of all costs associated with 
the direct provision of the treatment (-^.e., excluding project planning 
costs). The average unit cost for student training ranged from $0 at 
Wisconsin (all training costs were born by the participating schools as 
part of their regular teaching responsibilities) to $25.53 at Indiana 
University. However, comparing average cost per hour of training 
across projects may be unfair because of the variations in the 
intensity of the training and the number of students trained. To 
provide a more accurate comparison, the analysis should compute the 
average cost per hour of training per student trained . The per hour 
per student costs range from a low of $0.00 for the University of 
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Wisconsin to $.89 at Skyline Comtnunlty College, other projects grouped 
between $.01 and $.06 per hour. Skyline's costs are high because so 
few students were hired Into the program by the Toyota dealers and the 
program operated at level far lower than originally planned. 

The fourth cost issue was the service cost per unit of outcome. 
As In calculating average unit costs, the total treatment cost here Is 
the sum of all costs associated with providing the treatment. Service 
costs ranged from a low of $0 at University of Wisconsin to $19,266 at 
Skyline Consminlty College. Again, Skyl1ne*s costs are out of 
proportion with the other projects because the project trained fewer 
students than originally planned. 
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B. Recormnended Changes i n the Study Design for the $gCPP^ Ygar 



Based on the findings and experiences of the first year of the 
study, the study team reconnnends that the grantees to be site visited 
be selected based on the type of partnership and numbers of 
disadvantaged students served rather than on the evaluability of their 
application or the intensity of their services. The study team will 
use a four step process for selecting these sites. First, the 30 
FY 1989 projects will be categorized according to the type of 
partnership isipleniented. The types may Include; 1) partner as 
customer; 2) partner as advisory committee; 3) partner as supplier of 
resources such as equipment or job placements; 4) partner in sharing 
the delivery of instruction and services; or 5) partner as a developer 
seeking assistance. The study team will categorize the projects based 
on information from the mail survey rather than information contained 
in the original grant application. Several grantees admitted to the 
study team that they had not put much effort into preparing their 
application and were "surprised" to have been awarded grants. Thus, 
applications may reflect the applicant's perception of their chance to 
win rather than their true potential to conduct a successful 
demonstration. Furthermore, the quality of the grant application may 
also be a function of the grantees grant application writer rather than 
the capabilities of the grantee staff to manage a project. The sites 
to be visited would be selected after the mail survey has been 
completed. 

Second, the projects will be ranked by the six members of tha 
advisory panel according to the degree uf innovation in the partnership 
arrangenient. Projects showing the greatest amount of innovation 
relative to other projects known to the panel members would be ranked 
highest. The projects will be ranked using information from the mail 
survey and from the grant application. 

Third, the three most Innovative types of partnerships within each 
of the five partnerships categories will be selected for site visits. 
If there are more than three projects equally ranked at the top of the 
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category, three projects will be selected to represent three secondary 
types of Innovation: Innovation In curriculum, innovation In training 
techniques, or innovation in technology. 

Fourth, the remaining four sites will be selected based solely on 
the number of special population students being served. The 30 
projects v#ni be ranked according to the total number of special 
population students trained and/or placed into jobs, and the top four 
projects will receive site visits. 



